Open Grid Computing Environments Tutorial
Marlon Pierce, Suresh Marru, Gopi Kandaswamy, Gregor von Laszewski, and Tom Scavo
Tutorial Overview
Part 1: OGCE Portal (Marlon) o Introduction and overview o Portal Demo o How to build Grid portlets and gadgets Part 2: OGCE Workflow Tools (Suresh and Gopi) o Part A: Demo and overview o Part B: Hands on demo, deploying new applications Part 3: Web 2.0 for Grid Portals (Gregor)
Part 4: GridShib and Gateway security (Tom)
Goals of Tutorial
Goals o Review portal and gateway technologies old and new. o Demonstrate how to execute workflows and wrap TeraGrid applications. o Discuss security o Hear from TG Gateway community: what do you want? We have a lot of material to cover so we will sample many things. o You will have enough information to get going as a homework assignment. o Please also check with us at TG Please see Nancy’s “Gateway Parade” this afternoon.
Portal Demo
Demonstration of basic capabilities
Demo Follow-Up
What You Can Download
OGCE Portal
o Various portlet samplers
Web and Grid Services
o XRegistry: Web Service Registry o GFAC: Application Factory Service o Grid Portal Information Repository
Programming Libraries
o Java and JavaScript COG o Grid Tag Libraries and Beans (GTLAB) o Information Service Client APIs
Project Release Schedule
We have two major releases per year with intermediate updates. o TeraGrid (June) o Supercomputing (November) Focus of this release cycle: the workflow suite tools shown in Part 2. We do not currently package services and portlets together. o These are separate downloads.
Project Build and Test
Source Code Version Control and Building: We use Maven 2 and SourceForge’s SVN
o Anonymous checkouts o We are Open Source
Nightly Build Testing: We use the NMI Build and Test facility at UW
o Mac and Linux flavors
System Testing: We use Apache JMeter to verify portal builds function correctly.
o You can download this from us o You can also view the nightly dashboard
Project Information
Web site: http://www.collab-ogce.org Announcement Blog: http://collabogce.blogspot.com RSS/Atom Feeds: http://collabogce.blogspot.com/atom.xml o This is also embedded in the wiki as “News” Contact us: discuss@ogce.org
Building the Portal
Choosing a Host Computer
Select your host computer. o You will need a real IP address o VPN also OK o Know general Globus et al firewall issues.
I recommend turning firewalls and SELinux off for testing.
Operating Systems o We test with Mac OS X and many Linux flavors o Other Unix should be fine (assuming Java works) o We have not had any requests for Windows. Software Environment o The portal download comes with everything you need EXCEPT Java. You need the JDK, not the JRE. o We test with Java 1.5. Don’t use Java 1.4. The more memory, the better.
Downloading and Building the Portal
Download the portal from http://www.collab-ogce.org.
o Unpack in $HOME.
You should also download the Maven repository.
o This is optional but recommended
Edit ogce-portal-home/pom.xml
Run the command “mvn clean install”.
o Use “mvn –o clean install” to build offline.
You can also check out the latest from SVN. Check the “Nightly Build” dashboard to verify.
Build will typically take 5 minutes (new Mac) to 30 minutes (old Linux PC).
Editing pom.xml
You only need to edit the top level ogce-portal-only/pom.xml.
156.56.104.143 http://${portal.server.ip}:8080/ …. …. ${env.HOME}/ogce-portal-only
You only need to change
for local settings.
You MUST update the IP address. You may want to add or remove GRAM and GridFTP hosts. Update project.home if you are building someplace besides $HOME.
Rebuilding the Portal
Running “mvn clean install” will completely wipe out the existing installation.
If you want to rebuild a single component, just run the following command from the ogce-portal-only directory. o mvn clean install –f portlets/myportlet/pom.xml
OGCE Portal Project Layout
We follow standard Maven 2 conventions for project layouts for each component. o moduleName/src/main/webapp o moduleName/src/main/java We have the following modules o See top level pom.xml. o Each module is a separate directory and has its own pom.xml.
Module
Description
Portal_deploy
Containers
This is where your portlets are deployed into Tomcat.
Specific build instructions for containers. Currently we only have GridSphere 2.1.5 here. Controls look and feel for the container. Deploys COG jars into /shared/lib and applies Tomcat customconfigs.
Lnf_portal Global-config
Certificates Portlets
CA certs. Includes download of all TG certs. Each portlet has its own module.
Developing Grid User Interface Components
Portlets, gadgets, etc.
Under the Hood: Building Components
OGCE portlets use the Java COG Abstraction Layer to access Grid services. o Provides a buffer over different Grid toolkits and versions. o Provides ability to compose graph-based workflows. o Tutorial in itself Portlets can be built with our Apache Velocity bridge. o “Template” or “steal this code” approach. We have also developed Grid Tag Libraries and Beans (GTLAB) to simplify this process.
GTLAB Features
Extends Java Server Faces. o Tag components wrap major COG Abstraction Layer features Allows you to do standalone development. Use JSF portlet bridge to convert into portlets
No new coding, just change, add config files and jars.
Or forget about portlets and o Convert into Google Gadgets o Develop as a Facebook application o Etc.
Installing GTLAB
You need to install the OGCE portal o GTLAB depends on many shared jars.
Download the release code from www.collab-ogce.org or check out from SVN: o svn checkout https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB Edit GTLAB/jsf_standalone/pom.xml o should point to your OGCE portal server.
Edit GTLAB/jsf_standalone/src/main/resources/resources.properti es. Run “mvn clean install” in GTLAB/jsf_standalone/
Levels of Abstraction in GTLAB code
You can use prepackaged gadgets/portlets. You can mix and match repeatablesections of tags via “include” statements. You can develop new applications using existing tag libraries. You can hack the code
Note JSF is compatible with JSP, so you can mix and match. And you can embed JavaScript
o YUI, Scriptaculous user interface goodies.
o Google and Facebook APIs.
MyProxy Example
Show the Proxy Credential
Executing Multiple Task
GTLAB Example MyProxyExample
Description Get a credential and store in OGCE global proxy store.
JobSubmitExamples
Dashboard Example DirectoryBrowser MultipleTaskExample TaskMonitorExample
Various examples showing how to submit a grid job and get output.
Runs showq on Big Red and formats output. Displays and downloads files on a remote host via GridFTP Shows how to couple file transfers, operations, and View task state and metadata.
These will work as gadgets, portlets, or standalone applications
Making GTLAB Portlets
Do all development in jsf_standalone. When you are ready to convert: o cd GTLAB/transition o mvn clean process-resources o cd GTLAB/portlets o mvn clean install You will need to manually edit portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/ That’s it.
Making GTLAB Google Gadgets
A Gadget is a standalone application that is integrated into your personal iGoogle display. o They can run on your Web server. o You can use SSL/HTTPS Google Gadgets have two versions o http: these use Google’s JavaScript APIs o url: these wrap external applications with IFrames. Limitations: o Not much real estate o Cookies manage state on gadgets from the same server but need a way to integrate multiple independent servers.
Some GTLAB Gadgets
Example Gadget Config File
Save this as MyProxy.xml and place in a Web accessible place.
Some Web 2.0 Resources
See recent tutorials by Geoffrey Fox and me: o http://grids.ucs.indiana.edu/ptliupages/presentations /Web2.0Tutorial.ppt o http://grids.ucs.indiana.edu/ptliupages/presentations /CTSTechFutures_May19-08.pptx o Includes material on Google Gdata, Facebook API, Open Social, Microformats, etc. See also full examples at http://communitygrids.blogspot.com
TeraGrid Information Services Portlets and Libraries
Slides courtesy of Maytal Dahan
TG Information Services
What is TG Information Services? o Services that publish metadata about TeraGridd
TG Software, TG resources, TG Services
o Software and applications can query TG Info services for latest information Existing TG Info Services: o Currently we are relying on the REST interface of TG Info Services
TG Also has WebMDS interfaces
o Service includes CTSS4 information
TG Info Services
Using the REST Interfaces we plan to: o Develop JavaScript Libraries that automatically query and return service information o Libraries will handle all the querying and xml parsing necessary and return objects o API will include the ability to (and more)
getTGSites() – Get a list of all TG sites getResoures() – Get list of resources at a site getServices() – Get list of TG resources
Sample Interfaces
Talks, Demos, and BOFs
Monday Tuesday Tuesday 1:00 pm 11:00 am 1:30 pm Science Gateways on Parade (Nancy Wilkins-Diehr and a cast of 1000’s) TALK: TeraGrid Open Life Science Gateway (Wenjun Wu) TALK: Engaging with the LEAD Science Gateway Project: Lessons Learned in Successfully Deploying Complex System Solutions on TeraGrid TALK: Reliable and Scalable Job Submission: LEAD Science Gateway's Testing and Experiences with WS GRAM on TeraGrid Resources TALK: The LEAD-WxChallenge Pilot Project: The Potential of GridEnabled Learning BOF: TeraGrid User Portal BOF (Maytal Dahan) TALK: Workflow Infrastructure for Multi-scale Science Gateways TALK: Increasing TeraGrid User Productivity through Integration of Information and Interactive Services (Maytal Dahan) TALK: On-Demand Severe Weather Forecasts Using TeraGrid via the LEAD Portal POSTER/DEMO: TeraGrid Open Life Science Gateway POSTER/DEMO: Fault Tolerance and Recovery of Scientific Workflows in LEAD (Gopi)
35
Tuesday Tuesday Tuesday Wednesday Wednesday Wednesday Wednesday Wednesday
3:30 pm 3:30 pm 5:30 pm 10:30 am 2:00 pm 4:00 pm 6:30 pm 6:30 pm