Docstoc

Open Grid Computing Environments Tutorial

Document Sample
Open Grid Computing Environments Tutorial Powered By Docstoc
					Open Grid Computing
Environments Tutorial
Marlon Pierce, Suresh Marru, Gopi Kandaswamy, Gregor von
Laszewski, and Tom Scavo
Tutorial Overview
 Part 1: OGCE Portal (Marlon)
  o Introduction and overview
  o Portal Demo
  o How to build Grid portlets and gadgets

 Part 2: OGCE Workflow Tools (Suresh and Gopi)
  o Part A: Demo and overview
  o Part B: Hands on demo, deploying new applications

 Part 3: Web 2.0 for Grid Portals (Gregor)

 Part 4: GridShib and Gateway security (Tom)
Goals of Tutorial
 Goals
  o Review portal and gateway technologies old and new.
  o Demonstrate how to execute workflows and wrap TeraGrid
    applications.
  o Discuss security
  o Hear from TG Gateway community: what do you want?
 We have a lot of material to cover so we will sample many
  things.
  o You will have enough information to get going as a
    homework assignment.
  o Please also check with us at TG08
 Please see Nancy’s “Gateway Parade” this afternoon.
Slides and Demo Site
 Tutorial slides are available from http://www.collab-ogce.org

 We run a permanent demo portal at
  https://community.ucs.indiana.edu:8443/gridsphere/
  o Also aliased as
    https://ogceportal.iu.teragrid.org:8443/gridsphere

 Demo services are also available from Community
  o [NEED URLS]

 Accounts train30-train49 have been created for the workshop.
  Password is the same as the account name.

 We also have training accounts with same names that can be
  used to retrieve TG proxy credentials.
Portal and Gadget Demos
Demonstration of basic capabilities
Getting the Code
What You Can Download
 OGCE Portal
  o Various portlet samplers

 Web and Grid Services
  o XRegistry: Web Service
    Registry
  o GFAC: Application Factory
    Service
  o Grid Portal Information
    Repository

 Programming Libraries
  o Java and JavaScript COG
  o Grid Tag Libraries and Beans
    (GTLAB)
  o Information Service Client
    APIs
Project Release Schedule
 We have two major releases per year with
  intermediate updates.
  o TeraGrid (June)
  o Supercomputing (November)
 Focus of this release cycle: the workflow suite
  tools shown in Part 2.
 We do not currently package services and
  portlets together.
  o These are separate downloads.
Project Build and Test
 Source Code Version Control
  and Building: We use Maven 2
  and SourceForge’s SVN
  o Anonymous checkouts
  o We are Open Source

 Nightly Build Testing: We use
  the NMI Build and Test facility
  at UW
  o Mac and Linux flavors

 System Testing: We use Apache
  JMeter to verify portal builds
  function correctly.
  o You can download this from us
  o You can also view the nightly
    dashboard
Project Information
 Web site: http://www.collab-ogce.org
 Announcement Blog: http://collab-
  ogce.blogspot.com
 RSS/Atom Feeds: http://collab-
  ogce.blogspot.com/atom.xml
  o This is also embedded in the wiki as “News”
 Contact us: discuss@ogce.org
Building the Portal
Choosing a Host Computer
 Select your host computer.
  o You will need a real IP address
  o VPN also OK (?)
  o Know general Globus et al firewall issues.
     I recommend turning firewalls and SELinux off for development.

 Operating Systems
  o We test with Mac OS X and many Linux flavors
  o Other Unix should be fine (assuming Java works)
  o We have not had any requests for Windows.
 Software Environment
  o The portal download comes with everything you need EXCEPT
    Java. You need the JDK, not the JRE.
  o We test with Java 1.5. Don’t use Java 1.4.
 The more memory, the better.
Downloading and Building the Portal
 Download the portal from
  http://www.collab-ogce.org.
  o Unpack in $HOME.

 You should also download the
  Maven repository.
  o This is optional but
    recommended

 Edit ogce-portal-
  home/pom.xml
 Run the command “mvn clean
  install”.
  o Use “mvn –o clean install” to   You can also check out the latest
    build offline.                  code from SVN. Check the
                                    “Nightly Build” dashboard to
 Build will typically take 5
  minutes (new Mac) to 30+          verify.
  minutes (old Linux PC).
Practical Considerations
 Increase your JVM Memory
  o export JAVA_OPTS="-server -Xms512m -Xmx1024m -
    XX:MaxPermSize=256m”
  o Xmx2048m, 4096m, etc if you can.

 Monitor catalina.out for problems
  o tail –f catalina.out

 Make sure that Tomcat shuts down
  o Kill if necessary

 NFS can really slow down building
  o Use /usr/local/ if you have problems
                                     <properties>
Editing pom.xml                       <portal.server.ip>
                                       156.56.104.143
                                      </portal.server.ip>
 You only need to edit the top
  level ogce-portal-                  <host.base.url>
  only/pom.xml.                         http://${portal.server.ip}:8080/
                                      </host.base.url>
 You only need to change             <gridftp.host.names>
  <properties> for local settings.     ….
                                      </gridftp.host.names>
 You MUST update the IP
  address.                            <gram.host.names>
                                       ….
 You may want to add or              </gram.host.names>
  remove GRAM and GridFTP
  hosts.                              <project.home>
                                       ${env.HOME}/ogce-portal-only
 Update project.home if you          </project.home>
  are building someplace besides
  $HOME.                             </properties>
Rebuilding the Portal
 Running “mvn clean install” will completely wipe
  out the existing installation.
  o You may not want to do this.
 If you want to rebuild individual components,
  just run the following command from the ogce-
  portal-only directory.
  o mvn clean install –f
     portlets/myportlet/pom.xml
 You can do this for all <modules> in ogce-portal-
  deploy/pom.xml
OGCE Portal Project Layout
 We follow standard Maven 2 conventions
  for project layouts for each component.
  o moduleName/src/main/webapp
  o moduleName/src/main/java
 Modules are listed on next slide
  o See ogce-portal-only/pom.xml.
  o Each module is a separate directory and
    has its own pom.xml.
Module          Description
Portal_deploy   This is where your portlets are
                deployed into Tomcat.

Containers      Specific build instructions for
                containers. Currently we only have
                GridSphere 2.1.5 here.

Lnf_portal      Controls look and feel for the
                container.

Global-config   Deploys COG jars into /shared/lib and
                applies Tomcat custom configs.


Certificates    CA certs. Includes download of all TG
                certs.

Portlets        Each portlet has its own module.
Developing Grid User
Interface Components
Portlets, gadgets, etc.
Under the Hood: Building Components
 OGCE portlets use the Java COG Abstraction Layer to
  access Grid services.
  o Provides a buffer over different Grid toolkits and
    versions.
  o Provides ability to compose graph-based workflows.
  o Tutorial in itself

 Portlets can be built with our Apache Velocity bridge.
  o “Template” or “steal this code” approach.

 We have also developed Grid Tag Libraries and Beans
  (GTLAB) to simplify this process.
GTLAB Features
 Extends Java Server Faces.
  o Tag components wrap major COG Abstraction Layer
    features
 Allows you to do standalone development.
 Use JSF portlet bridge to convert into portlets
     No new coding, just change, add config files and jars.

 Or forget about portlets and
  o Convert into Google Gadgets
  o Develop as a Facebook application
  o Etc.
Installing GTLAB
 You need to install the OGCE portal
  o GTLAB depends on many shared jars.

 Download the release code from www.collab-ogce.org or
  check out from SVN:
  o svn checkout
    https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB

 Edit GTLAB/jsf_standalone/pom.xml
  o <catalina.home> should point to your OGCE portal server.

 Edit
  GTLAB/jsf_standalone/src/main/resources/resources.properti
  es.

 Run “mvn clean install” in GTLAB/jsf_standalone/
Levels of Abstraction in GTLAB code
 You can use prepackaged gadgets/portlets.
 You can mix and match repeatable sections of
 tags via “include” statements.
 You can develop new applications using
 existing tag libraries.
 You can hack the code
  Note JSF is compatible with JSP, so you can mix and
   match.
  And you can embed JavaScript
   o YUI, Scriptaculous user interface goodies.
   o Google and Facebook APIs.
                      MyProxy Example
<h:form id="form">
  <h:outputText value="Username:"/>
  <h:inputText value="#{resource.username}" required="true"/>

 <h:outputText value="Password: "/>
 <h:inputSecret value="#{resource.password}" required="true"/>

 <h:outputText value="MyProxy Server: "/>
 <h:inputText value="#{resource.myproxyHostname}" required="true"/>

  <o:submit id="submit" value="Submit" binding="#{builder.body}"
              action="#{builder.junkAction}" >
    <o:multitask id="multi" persistent="true" taskname="#{resource.taskname}">
     <o:myproxy id="mypr" hostname="#{resource.myproxyHostname}"
              lifetime="2" password="#{resource.password}" port="7512"
              username="#{resource.username}"/>
   </o:multitask>
 </o:submit>
</h:form>
           Show the Proxy Credential
<h:dataTable value="#{myproxy.storedProxyList}" border="1"
             binding="#{myproxy.storedProxyTable}"
                      var="proxyEntry"
                      id="proxyTable">
     <h:column>
       <f:facet name="header">
        <h:outputFormat id="subject" escape="false” value="Subject"/>
       </f:facet>
       <h:outputText id="proxySubject" value="#{proxyEntry.subject}"/>
     </h:column>

     <h:column>
       <f:facet name="header">
        <h:outputFormat id="Remove" escape="false” value="Delete"/>
       </f:facet>
       <h:commandButton id="deleteProxy" value="Remove”
                 actionListener="#{myproxy.deleteCredential}"/>
     </h:column>
</h:dataTable>
            Executing Multiple Task
<o:submit id="submit" value="Submit" binding="#{builder.body}"
           action="#{builder.junkAction}" >
   <o:multitask id="multi" persistent="true"
           taskname="#{resource.taskname}">
     <o:jobsubmit id="js" myfaces="true"
           arguments="#{resource.arguments}"
           executable="#{resource.executable}"
           hostname="#{resource.hostname}"
           provider="#{resource.provider}"
           stdout="#{resource.stdout}"/>
     <o:filetransfer id="ft" myfaces="true"
               from="#{resource.from}" to="#{resource.to}" />
     <o:dependency id="dep" task="ft" dependsOn="js"/>
   </o:multitask>
</o:submit>
GTLAB Example         Description
MyProxyExample        Get a credential and store in OGCE
                      global proxy store.
JobSubmitExamples     Various examples showing how to
                      submit a grid job and get output.
Dashboard Example     Runs showq on Big Red and formats
                      output.
DirectoryBrowser      Displays and downloads files on a
                      remote host via GridFTP
MultipleTaskExample   Shows how to couple file transfers,
                      operations, and
TaskMonitorExample    View task state and metadata.

         These will work as gadgets, portlets,
         or standalone applications
Making GTLAB Gadgets into Portlets
 Do all development in jsf_standalone.

 When you are ready to convert:
  o cd GTLAB/transition
  o mvn clean process-resources
  o cd GTLAB/portlets
  o mvn clean install

 You will need to manually edit portlet.xml, group.xml,
  and layout.xml in
  GTLAB/portlet/src/main/webapp/WEB-INF/

 That’s it.
Making GTLAB Google Gadgets
 A Gadget is a standalone application that is integrated
  into your personal iGoogle display.
  o They can run on your Web server.
  o You can use SSL/HTTPS
 Google Gadgets have two versions
  o http: these use Google’s JavaScript APIs
  o url: these wrap external applications with IFrames.
 Limitations:
  o Not much real estate (but location, location, location)
  o Cookies manage state on gadgets from the same server
    but we need a way to integrate multiple independent
    servers.
Some GTLAB Gadgets
         Example Gadget Config File
<?xml version="1.0" encoding="UTF-8" ?>
<Module>
 <ModulePrefs title="MyProxy Gadget Example"
                scrolling="true"
                height="500"/>
 <Content type="url"
   href="http://…/:8080/GTLAB/examples/MyProxyExample.jsf">
 </Content>
</Module>


         Save this as MyProxy.xml and place
         in a Web accessible place.
Some Web 2.0 Resources
 See recent tutorials by Geoffrey Fox and me:
  o http://grids.ucs.indiana.edu/ptliupages/prese
    ntations/Web2.0Tutorial.ppt
  o http://grids.ucs.indiana.edu/ptliupages/prese
    ntations/CTSTechFutures_May19-08.pptx
  o Includes material on Google Gdata, Facebook
    API, Open Social, Microformats, etc.
 See also full examples at
  http://communitygrids.blogspot.com
TeraGrid Information
Services: Portlets and
Gadget Libraries
Slides courtesy of Maytal Dahan
TG Information Services
 What are TG Information Services?
  o Services that publish metadata about TeraGridd
    TG Software, TG resources, TG Services
 o Software and applications can query TG Info
   services for latest information
 Existing TG Info Services:
  o Currently we are relying on the REST interface
    of TG Info Services
    TG Also has WebMDS interfaces
 o Service includes CTSS4 information
Libraries for TG Info Services
 Using the REST Interfaces we are
  o Developing JavaScript Libraries that automatically
    query and return service information
  o Handling all the querying and xml parsing
    necessary and return objects through libraries
  o API will include the ability to (and more)
    getTGSites() – Get a list of all TG sites
    getResoures() – Get list of resources at a site
    getServices() – Get list of TG resources

 Late summer availability
 Demo portlets available now
Sample Portlet Interfaces
            Talks, Demos, and BOFs                                                             37


Monday      1:00 pm    Science Gateways on Parade (Nancy Wilkins-Diehr and a cast of 1000’s)

Tuesday     11:00 am   TALK: TeraGrid Open Life Science Gateway (Wenjun Wu)

Tuesday     1:30 pm    TALK: Engaging with the LEAD Science Gateway Project: Lessons
                       Learned in Successfully Deploying Complex System Solutions on
                       TeraGrid

Tuesday     3:30 pm    TALK: Reliable and Scalable Job Submission: LEAD Science Gateway's
                       Testing and Experiences with WS GRAM on TeraGrid Resources
Tuesday     3:30 pm    TALK: The LEAD-WxChallenge Pilot Project: The Potential of Grid-
                       Enabled Learning
Tuesday     5:30 pm    BOF: TeraGrid User Portal BOF (Maytal Dahan)
Wednesday   10:30 am   TALK: Workflow Infrastructure for Multi-scale Science Gateways

Wednesday   2:00 pm    TALK: Increasing TeraGrid User Productivity through Integration of
                       Information and Interactive Services (Maytal Dahan)
Wednesday   4:00 pm    TALK: On-Demand Severe Weather Forecasts Using TeraGrid via the
                       LEAD Portal
Wednesday   6:30 pm    POSTER/DEMO: TeraGrid Open Life Science Gateway
Wednesday   6:30 pm    POSTER/DEMO: Fault Tolerance and Recovery of Scientific Workflows
                       in LEAD (Gopi)

				
DOCUMENT INFO