Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

SRM in GGF

VIEWS: 6 PAGES: 16

									             SRM-Lite:
 overcoming the firewall barrier
  for large scale file replication

           Arie Shoshani
             Alex Sim
Lawrence Berkeley National Laboratory


              April, 2007


                                        1
                     First, some background:
                          What are SRMs

• Storage Resource Managers (SRMs) are middleware components
  whose function is to provide:
   • dynamic space allocation AND file management in spaces
       • for storage components on the local or wide-area network
       • Based on a common standard


                     client/user applications

         SRM         SRM/      SRM/              SRM       SRM
         (DPM)       dCache    CASTOR           (StoRM)   (BeStMan)



        Unix-based
           Disk                                                       Unix-based
                                                                         Disk
          Pools                                 GPFS                    Pools
                     dCache    CASTOR

                                        CCLRC
                                        RAL




 Examples of some storage systems currently supported by SRMs                      2
                   SRM Functional Concepts

• Manage Spaces dynamically
   • Reservation, lifetime
   • Negotiation
• Manage files in spaces
   •   Request to put files in spaces
   •   Request to get files from spaces
   •   Lifetime, pining of files, release of files
   •   No logical name space management (done by replica location services)
• Access remote sites for files
   • Bring files from other sites and SRMs as requested
   • Use existing transport services (GridFTP, https, …)
   • Transfer protocol negotiation
• Manage multi-file requests
   • Manage request queues
   • Manage caches
   • Manage garbage collection
• Directory Management
   • Uxix semantics: srmLs, srmMkdir, srmMv, srmRm, srmRmdir

                                                                              3
Example Use of SRMs in Earth Science Grid
        (in production for 3 years)
3100 users                                                                                              120 TBs managed

                               LBNL
         HPSS
                            disk
   High Performance
    Storage System                                                                                                     ANL
                                                                                                                       CAS
                                                                                                           Community Authorization Services

         HRM
                              gridFTP
                                           NCAR openDAPg                   gridFTP
                                                                           Striped
   Storage Resource                                       server
                               server                                       server
      Management

                                                                                         MyProxy
                                             Tomcat servlet engine                        server


    disk         LLNL                            MCS client    MyProxy client

                                                                     DRM                   CAS client
                                                 RLS client    Storage Resource
                                                                  Management

          DRM
    Storage Resource          gridFTP                                                  GRAM
       Management              server                                                gatekeeper
                                                     gridFTP
                                           gridFTP
                                                      server                                                         ORNL
                                                                                                                            HRM
                      ISI                                       HRM
                                                                                             gridFTP     gridFTP
                                                                                                          server
                                                                                                                      Storage Resource
                                                                                                                         Management
             MCS                                          Storage Resource
                                    SOAP
 Metadata Cataloguing Services                               Management


             RLS                    RMI                                                                                        HPSS
   Replica Location Services                                              MSS                                 disk       High Performance
                                                       disk
                                                                   Mass Storage System                                    Storage System


 LBNL’s SRMs inter-communicate between several sites and main portal site at NCAR
            to provide storage management and multi-file movement                                                                             4
            DataMover: SRMs used to provide large scale
                robust data streaming between sites

•   Problem: move thousands of
    files robustly                                            Anywhere
     •    Takes many hours
     •    Need error recovery                                  DataMover
            • Mass storage systems
              failures
            • Network failures
•   Solution: Use Storage Resource                       SRM-COPY                      Get list
    Managers (SRMs)                                      (thousands of files)          of files
     •    File streaming paradigm       NERSC                                                                BNL
     •    By reserving and releasing
          storage space automatically                           SRM-GET (one file at a time)

                                              SRM                                                     SRM
                                          (performs writes)                                       (performs reads)
•   Problem: too slow                                           GridFTP GET (pull mode)

•   Solution:
     •    in GridFTP
            •   Use parallel streams                   Disk                                        Disk
            •   Use large FTP windows                 Cache                                       Cache
     •    Pre-stage files from MSS
                                                                    Network transfer
     •    Use concurrent transfers

                                            archive files                                            stage files




         Example setup for STAR high-energy-physics experiment
                                                                                                                     5
                             SRM-Lite


• Goal 1: automate file movement behind a firewall
   •   a client program
   •   to automate movement of multiple files
   •   to/from client’s directory to a remote site
   •   given a OTP firewall at one site
   •   Support entire directory transfers
   •   Recover from mid-transfer interruption and machine failure
• Goal 2: pull files into user’s workstation
   •   Use SRM-Lite by users to download files into their workstations
   •   Using various transfer protocols (GridFTP, bbcp, https, …)
   •   Have a GUI that shows transfer progress
   •   Or have a command line
   •   Support entire directory transfers
   •   Support suspend/resume operations (e.g. on laptops)


                                                                         6
   SRM-Lite: a client program to automate movement of multiple
      files to/from client’s directory to a remote SSH server
                   given a OTP firewall at one site

                                                   Use Case A:
• Process Steps                          OTP firewall at local site (ORNL),
   • Login to ORNL using OTP             SSH server at remote site (NERSC)

   • At ORNL invoke SRM-Lite
   • User composes XML input
     file, srmlite.xml for selected                                           OTP
                                                                              Login
     files/directories to copy from/to
                                                                                          ORNL
     another site                            NERSC

   • Or, user gives command line                                                 SRM-
                                            SSH Server          SSH
     option for a selected                                     Request            Lite
     file/directory
   • SRM-Lite uses srmlite.xml or          Local Commands
     command line input
     to automatically
                                                                                      srmlite.xml
      • Push/Pull files to/from NERSC                         GridFTP/FTP/
                                                                  SCP
                                                      Disk                        Disk
      • Use multiple threads for                      Cache    transfers          Cache

        concurrent transfers               HPSS




                                                                                                    7
                    Scenario: one end has SRM,
                The other end has a firewall, use SRM

                                                   Use Case B:
• Process Steps                           OTP firewall at local site (ORNL),
   • Login to ORNL using OTP             SRM server at remote site (NERSC)
   • At ORNL invoke SRM-Lite
   • User composes XML input
     file, srmlite.txt for selected                                            OTP
     files/directories to copy over                                            Login
     to/from another SRM controlled                                                        ORNL
                                              NERSC
     storage system
   • Or, user gives command line                                                  SRM-
     option for a selected                                                         Lite
     file/directory
   • SRM-Lite uses srmlite.xml or
     command line input                           SRM
     to automatically
      • Push/Pull files to/from SRM at                          GridFTP/FTP/
                                                                                       srmlite.xml
        NERSC                                           Disk
                                                                    SCP
                                                                                   Disk
                                                                 transfers
      • Use multiple threads for                        Cache                      Cache

        concurrent transfers               HPSS




                                                                                                     8
                   Scenario: one end has SRM,
         The other end has a firewall, use either SSH/SRM,

                                                   Use Case C:
• Process Steps                           OTP firewall at local site (ORNL),
   • Login to ORNL using OTP            SRM/SSH server at remote site (NERSC)
   • At ORNL invoke SRM-Lite
   • User composes XML input
     file, srmlite.xml for selected                                                OTP
     files/directories to copy over                                                Login
     to another site                                                                           ORNL
                                                 NERSC
   • Or, user gives command line
     option for a selected                                            SSH             SRM-
     file/directory                             SSH Server
                                                                     Request           Lite
   • SRM-Lite uses srmlite.xml or
     command line input
     to automatically                                 SRM
      • Push/Pull files to/from NERSC
        using either SSH or SRM                                     GridFTP/FTP/
                                                                                           srmlite.txt
      • Use multiple threads for                            Disk
                                                                        SCP
                                                                                       Disk
        concurrent transfers                                Cache    transfers         Cache
                                               HPSS




                                                                                                         9
                  Scenario: both ends have SRMs,
         both ends have a firewall, use SRM-lite on both ends,
               Use SSH to invoke SRM-Lite at other end

                                                       Use Case D:
•   Process Steps
                                          OTP firewall at both local site (ORNL),
    • Login to ORNL using OTP                   and remote site (NERSC),
    • Create a OTP SSH tunnel to                Use SRM-lite at both ends
    • NERSC
    • User composes XML input
      file, srmlite.txt for selected                                                   OTP
                                                                                       Login
      files/directories to copy over
      to another site                         NERSC                                            ORNL
    • Or, user gives command line                                     OTP
                                                                                    SSH-Tunneling
      option for a selected                                        SSH Tunnel
      file/directory                              SRM-              SRM                  SRM-
    • SRM-Lite uses srmlite.txt or                 Lite            Request                Lite
      command line input, through
                                                                                                    srmlite.txt
      SSH tunnel to automatically
        • Communicate with SRM-lite at            SRM
          other end                                                                      SRM
        • Push/Pull files to/from NERSC
          using SRM                                                  SCP
                                                          Disk
        • Use multiple threads for                        Cache
                                                                   transfers
                                                                                      Disk
          concurrent transfers                                                        Cache
                                           HPSS
        • Can use SCP only                                                                          HPSS




                                                                                                            10
                      SRM-Lite: Status


• SRM-Lite is developed
• Available from: http://datagrid.lbl.gov/srmlite/
• Tested with GridFTP, SCP, HTTPS, HTTP
• Tested with large number of files
• Tested behind a firewall
• Access from local SRMs that acess HPSS
  – not tested yet
• Access between two filewalled system
  – not developed yet




                                                     11
                        SRM-Lite: GUI

• GUI was developed when used at user’s site
  (Linux, PC, MAC), called DataMover-lite (DML)
• Available from: http://datagrid.lbl.gov/dml/
• Example GUI
  screen
• Shows info on:
  completed, active,
  and pending
  transfers
• Also, file sizes,
  transfer times,
  transfer speed




                                                  12
Extra Slides




               13
      SRM
     (StoRM)                Storage Resource Managers

• SRMs are middleware components whose function
  is to provide:
   • dynamic space allocation AND file management in spaces
   • for storage components on the local or wide-area network
   • Based on a common standard


                                             client/user applications

  SRM/          SRM           SRM        SRM/      SRM/    SRM/        SRM       SRM         SRM          SRM
  L-Store       (DPM)       (Jlab-SRM)   dCache   CASTOR   CASTOR     (StoRM)   (BeStMan)   (BeStMan)   (BeStMan)




               Unix-based
                 disks      JASMine                                   GPFS                  MSS         Unix-based
                                         dCache                                                           disks
                                                  CASTOR   CASTOR
                                                              CCLRC
                                                              RAL




   Examples of storage systems currently supported by SRMs                                                           14
        DataMover-Lite use in ESG: a client program used
   to automate movement of multiple files to client’s directory


• Process Steps
  • User downloads DataMoverLite
  • User goes to portal, select files
  • Portal gets ALL files into                              datamover.txt
    SRM disk
  • Portal generates XML input file,      DataPortal
                                           NCAR                             User’s machine
    datamover.txt, for user
    selected files                                             request
                                         ESG Portal                           User’s
  • DML uses datamover.txt to                                                 browser
    automatically
     • get files, and                                                        DataMover
     • release files after move               SRM                               Lite
       completes successfully
                                                             GridFTP/FTP/
                                                             HTTP/HTTPS
                                                    Disk                        Disk
                                                    Cache     transfers         Cache
                                        MSS




                                                                                             15
Another example of DML GUI




                             16

								
To top