Development of the Perl CoG

Document Sample
Development of the Perl CoG Powered By Docstoc
					Development of the Perl CoG



             Stephen Mock
  Computational Science Portals Group,
    San Diego Supercomputer Center



             Presented at
           EuroGlobus 2001
               June 20
           Marina di Ugento
             Lecce, Italy
                    Outline


•   What is a CoG?
•   Why PERL
•   Evolution of the PERL CoG
•   Modules & Examples
•   Future Work
•   Conclusion
•   Links and email
•   Acknowledgements




                         2
                           CoGs

• CoG = Commodity Grid Toolkit
• What is a CoG?
  – Kits that provide users and developers in a virtual
    organization with seamless access to Grid Services using
    different programming languages


• Current CoG Projects:
  –   Java (GPDK is based on the Java CoG)
  –   Python
  –   CORBA
  –   PERL (new!)
  –   Perhaps the GRB will release a C based CoG?




                                3
                        Why PERL

• PERL CoG driven by HotPage
   – User portal that gives information about computational
     resources within NPACI
   – Written in PERL since way before server-side java
     developed into a robust way to program web applications
   – PERL+ Globus was the most simple solution to add
     interactive access to the portal
      • Job submission and control
      • File and directory manipulation/transfer
• This led to the creation of the GridPort toolkit
   – Abstraction of the scripts used to add interactive access
   – Used to develop other portals which use grid services
   – Further abstraction of those scripts is leading to the
     PERL CoG modules


                                  4
HotPage Snapshot




         5
          PERL Programming Language

• PERL is a useful and versatile programming
  language
   – Adaptable for different uses and highly portable
   – Ubiquitous
   – Powerful for text manipulation
      • Parsing output
      • Generating html
   – Easy to use in a webserver environment
      • No extension to the webserver needed to run CGI/PERL
      • No need to recompile or reboot the server if a script
        changes
   – Familiar to many scientists
   – Object oriented or not, depending on your needs



                                 6
                     GridPort Toolkit

• Grid Portal Toolkit
   – Toolkit for developing grid aware portals in Perl
   – Written in PERL as libraries
   – Several portals running using GridPort software
      •   https://hotpage.npaci.edu and https://hotpage.paci.org
      •   https://gridport.npaci.edu/LAPK
      •   https://gridport.npaci.edu/Telescience
      •   https://gridport.npaci.edu/GAMESS
   – Allows authentication, batch job submission and control,
     file transfer, and interactive job/command submission




                                    7
GridPort Architecture




           8
             PERL CoG vs GridPort

GridPort
• GridPort is written as PERL libraries
• GridPort is written specifically for development of
  web applications
• However, GridPort is being rewritten with the
  PERL CoG
PERL CoG
• The PERL CoG is written as PERL modules
• The PERL CoG can be used by any PERL application
  (command-line scripts, system utilities, user
  applications)
• The PERL CoG provides better abstraction of the
  grid services
                            9
                  Perl CoG - Globus

• Some of the modules are wrappers to Globus
  functions
• Use installed Globus utilites
   –   Grid-proxy-init
   –   Globusrun
   –   Globus-job-submit
   –   Grid-info-search


• Plans to rewrite some of the modules in Pure Perl
   – No need to install globus utilities
   – Direct connections to the services like the globus
     gatekeeper and MDS



                                10
CoG Use




    11
                      The Modules

• Tentative first pass at the architecture
   – Already have changes that will be applied


• Completed Modules:
   –   Grid::Globus::RSL.pm
   –   Grid::Globus::Globusrun.pm
   –   Grid::Globus::GlobusJob.pm
   –   Grid::Info::Search.pm
   –   Grid::Myproxy.pm
   –   Grid::Proxy.pm




                                12
                  Grid::Globus::RSL.pm

• Module for generating Globus RSL strings
• API:
   – toString()
       • returns formatted rsl string with (‘s and &’s
   – setRSL()
       • sets the rsl string
   – getRSL()
       • prints non-formatted rsl string
   – clearRSL()
       • deletes the current rsl string
   – addAttribute()
       • adds an rsl attribute to the rsl string
   – deleteAttribute()
       • removes an attribute from the rsl
   – getAttribute()
       • returns the string associated with an attribute




                                          13
       Grid::Globus::RSL.pm Example

use Grid::Globus::RSL;

my $rsl = Grid::Globus::RSL->new();
if(defined($rsl))
{
   $rsl->addAttribute(“executable”,”/bin/ls”);
}

$generated_rsl = $rsl->toString();




                          14
           Grid::Globus::Globusrun.pm

• Module wrapper to the globusrun command
• Runs an interactive job and returns the output
• first module written
   – have many changes I would like to make to it.
• API:
   – globusrun()




                                15
   Grid::Globus::Globusrun.pm Example

use Grid::Globus::Globusrun;
use Grid::Globus::RSL;
my $myRSL = Grid::Globus::RSL->new();
$myRSL->addAttribute('count','1');
$myRSL->addAttribute('executable','/bin/ls');
$myRSL->addAttribute('arguments','-l');

my @output = globusrun(
           “$globusrun_arguments",
           "tfglobus.sdsc.edu",
           $myRSL->toString(),
           “$timeout"
           );



                         16
            Grid::Globus::GlobusJob.pm

• Module for submitting batch jobs via Globus
• Need a name change
• API:
   –   new()
   –   getStatus()
   –   getError()
   –   getJobid()
   –   setContact()
   –   setExecutable()
   –   setCpus()
   –   setQueue()
   –   setMaxtime()
   –   submit()
   –   cancel()
   –   getStderr/Stdout()


                                17
    Grid::Globus::GlobusJob.pm Example

use Grid::Globus::GlobusJob.pm;

my $job = Grid::Globus::GlobusJob->new();
$job->setContact($host);
$job->setExecutable($exe);
$job->setMaxtime($maxTime);
$job->setCpus($cpus);
$job->setQueue($queue);
$job->submit();

print “Job status is:” . $job->getStatus();
my @stdout = $job->getStdout();
if($impatient) {
   $job->cancel();
}


                             18
              Grid::Info::Search.pm

• Module for querying the MDS
• Currently uses grid-info-search utility, but will
  move to a pure perl implementation soon
• API:
   – new()
   – execute()
   – getResults()




                             19
      Grid::Info::Search.pm Example

use Grid::Info::Search;

my $gis = Grid::Info::Search->
                new(
                     $brach_point,
                     $filter,
                     $timeout
                    );
if(defined($gis)) {
   foreach($gis->getResults()) {
      print;
   }
}
else { die “Could not perform search: $!”; }



                          20
                   Myproxy.pm

• Module for using MyProxy:
• API:
  – myproxyInit()
  – myproxyDestroy()
  – myproxyGetDelegation()




                             21
         Grid::Myproxy.pm Example

use Grid::Myproxy;

$myp = Grid::Myproxy->new();
$myp->setMyproxyInit(
      "$username","$pem_passphrase",
      "$myproxy_passphrase","$server_name“
                     );

$myp->myproxyInit();
$myp->myproxyGetDelegation();

$res= $myp->getMyproxyResult();
$myp->myproxyDestroy();



                          22
                    Grid::Proxy.pm

• Module for dealing with user proxy files:
• API:
   – proxyInit()
      • Generate a user proxy file
   – proxyDestroy()
      • Delete a user proxy file
   – getInfo()
      • Get information about a proxy
          – Lifetime
          – Strength
          – User information




                                     23
         Grid::Proxy.pm Example

use Grid::Proxy;

my $results =
  proxyInit($pass,$cert,$key,$out,$hours,
  $limited,$bits,$cert_dir);
print "Proxy Init Results: $results\n";

my $destroy_results = proxyDestroy($out);




                     24
                       Future Work

• Pure PERL implementations
   – Eliminate the need to have globus client tools installed
   – Grid-info-search via the Net::LDAP module
   – Gram module so that we don’t have to use shell commands to run
     globus jobs
• Changes to the current architecture
• More modules
   – Configuration module
   – GASS modules
   – GSI modules
       • gsissh
       • gsiftp
   – Storage Resoure Broker (SRB) module
• Rewrite GridPort based on these modules
   – Simplify the code and installation


                                   25
               Conclusion

The PERL CoG provides users and developers
 of applications access to Grid Services
 using the PERL programming language




                      26
                     Links and Email

Input:
   – What do YOU want to see?
   – Find me here or email me
         • mock@sdsc.edu
Links:
   – GridPort - http://gridport.npaci.edu
   – HotPage – https://hotpage.npaci.edu
   – CoG – can be found at the GridPort web site soon




                                27
                  Acknowledgements

Module design and development:
   – Stephen Mock, mock@sdsc.edu
   – Maytal Dahan, mdahan@sdsc.edu


Input:
   –   Mary Thomas, mthomas@sdsc.edu
   –   Gregor von Laszewski, gregor@mcs.anl.gov
   –   Kurt Mueller, mueller@sdsc.edu
   –   Catherine Mills, cmills@sdsc.edu




                                28