Get documents about "NGS in the future emerging middleware
Document Sample
http://www.grid-support.ac.uk http://www.ngs.ac.uk
NGS in the future: emerging
middleware
Mike Mineter
mjm@nesc.ac.uk
http://www.nesc.ac.uk/ http://www.pparc.ac.uk/ http://www.eu-egee.org/
Goal of talk
• The NGS is running a production service
• Different middleware may be deployed in the
future.
• The talk seeks to outline some of the
possibilities
2
NGS middleware evolution
EGEE… Other software NGS
sources Software with proven UK,
Prototypes &
ETF capability & realistic
deployment Operations Campus
specifications
experience and other
„Gold‟ services grids
Feedback & future requirements
Engineering Task Force
Deployment/testing/advice
3
Outline
• Middleware currently being prepared for
deployment
– Resource broker
• Under assessment:
– gLite middleware from EGEE
– OMII
– GT4
4
Resource broker
• (This is NOT the SRB!!!)
• Current NGS middleware : Toolkits inviting
development of higher level services
• On the current NGS we have
– GRAM to submit jobs
– Information service to tell us what queues are busy
• The RB takes the work out of deciding where to
run a job
5
Major components
Replica
“User Input “sandbox”
DataSets info Catalogue
interface” Information
Output “sandbox”
Service
Resource
Broker
Job Submit Event
Author.
Job Query
&Authen.
Publish
Storage
Element
Logging & Computing
Book-keeping Job Status
Element
Resource broker
• Job Description Language file: describes
resources needed by a job
• Commands analogous to GT2:
– edg-job-submit <jdl filename>
– edg-job-status <dg-job-id>
– edg-job-get-output <dg-job-id>
7
Example
• edg-job-submit myjob.jdl
– Myjob.jdl
• JobType = “Normal”;
• Executable = "$(CMS)/exe/sum.exe";
• InputSandbox = {"/home/user/WP1testC","/home/file*”,
"/home/user/DATA/*"};
• OutputSandbox = {“sim.err”, “test.out”, “sim.log"};
• Requirements = other. GlueHostOperatingSystemName ==
“linux" &&
• other. GlueHostOperatingSystemRelease == "Red Hat 7.3“
&& other.GlueCEPolicyMaxCPUTime > 10000;
• Rank = other.GlueCEStateFreeCPUs;
8
More about the RB
• Developed by the European DataGrid project, EDG then
“hardened” by LCG, and now one of the sources for the
EGEE middleware
• Uses components of Condor
– matchmaker and Condor-G
• Try the GENIUS portal on GILDA
– GILDA is a dissemination grid running the LCG-2 middleware
– Demo site: https://grid-demo.ct.infn.it/
• And look at
http://lcg.web.cern.ch/LCG/
http://www.hep.ph.ic.ac.uk/e-
science/projects/demo/index.html
9
Resource broker -
summary
• The resource broker receives a job description
in JDL
• It chooses a batch queue for job submission,
using the information services
• Its an example of the higher services that can be
deployed for the NGS, built upon the current
toolkits
10
Outline
• Middleware currently being prepared for
deployment
– Resource broker
– (NGS portal – yesterday!)
• Under assessment:
– gLite middleware from EGEE
– OMII
– GT4
11
EGEE – towards e-infrastructure
Enabling Grids for E-sciencE
EGEE is building a large-scale
production grid service to: Collaboration
Operations, Support and
• Underpin research,
technology and public service
• Link with and build on
national, regional and Pan-European Grid
international initiatives
training
• Foster international
cooperation both in the
creation and the use of the e- Network
infrastructure infrastructure
& Resource
centres
INFSO-RI-508833 12
In the first 2 years EGEE will
Enabling Grids for E-sciencE
• Establish production quality sustained Grid
services
– 3000 users from at least 5 disciplines
– integrate 50 sites into a common
infrastructure
– offer 5 Petabytes (1015) storage
• Demonstrate a viable general process to
bring other scientific communities on board
• Propose a second phase in mid 2005 to take Pilot New
over EGEE in early 2006
INFSO-RI-508833 14
EGEE Organisation
Enabling Grids for E-sciencE
• 70 leading institutions in 27
countries, federated in regional Grids
• ~32 M Euros EU funding for first 2
years starting April 2004
(matching funds from partners)
• Leveraging national and regional grid
activities
• Promoting scientific partnership
outside EU
INFSO-RI-508833 15
Activities Definition
Enabling Grids for E-sciencE
• Network Activities
– NA1: Project Management
– NA2: Dissemination and Outreach
– NA3: User Training and Induction
– NA4: Application Identification and Support
– NA5: Policy and International Cooperation
• Service Activities
– SA1: Grid Support, Operation and Management
– SA2: Network Resource Provision
• Joint Research Activities
– JRA1: Middleware Reengineering + Integration
– JRA2: Quality Assurance
– JRA3: Security Emphasis in EGEE is on
– JRA4: Network Services Development operating a production
grid and supporting the
end-users
INFSO-RI-508833 16
gLite: Guiding Principles
Enabling Grids for E-sciencE
VDT EDG ...
• Service oriented approach
– Allow for multiple interoperable
implementations
• Lightweight (existing) services AliEn LCG ...
– Easily and quickly deployable
– Use existing services where
possible
Condor, EDG, Globus, LCG, …
• Portable
– Being built on Scientific Linux and • Co-existence with deployed
Windows infrastructure
• Security – Co-existence with LCG-2 and OSG
– Sites and Applications (US) are essential for the EGEE
• Performance/Scalability & Grid services
Resilience/Fault Tolerance • Site autonomy
– Comparable to deployed – Reduce dependence on „global,
infrastructure central‟ services
• Open source license
INFSO-RI-508833 17
gLite Services for Release 1
Enabling Grids for E-sciencE
JRA3 Grid Access UK
API
Service
CERN IT/CZ
Access Services
Authorization Information & Application
Auditing Monitoring Monitoring
Authentication Information &
Monitoring Services
Security Services
Metadata File & Replica Job Package
Catalog Catalog Accounting Provenance Manager
Storage Data Computing Workload
Element Management Site Proxy Element Management
Data Services Job Management Services
INFSO-RI-508833 18
Open Middleware
Infrastructure Institute
The slides that follow were selected and (in a
few cases) modified by Mike Mineter (NeSC)
from those presented in January 2005 at an
OMII training day
Steven Newhouse, Peter Henderson
Stephen Crouch & Karen Ng
Goal of this presentation: to raise awareness
of the OMII and its OMII_1 release
MM 19
Open Middleware
Infrastructure Institute
OMII goal: to be the source of open source grid
software
Institute of the University of Southampton
Utilise existing software and standards
Production focused software development
Integrate, test & document „a product‟
Focus on the user experience
Easy to install & use
Utilise existing software and standards
Provide a solid web service base for others to build on
20
Where does our software come
from?
Open Source Community
Tomcat, Axis, etc.,
Software Repository
Accept software contributions
Software deployed, tested & graded to provide
feedback
Managed Programme
Fill gaps to build a solid enabling infrastructure
Projects to bring research software to production
quality
21
Managed Programme
GridSAM (Job Submission & Monitoring service)
BPEL (Workflow service)
Grimoires (Registry service based on UDDI)
FIRMS (Reliable messaging)
FINS (Notification)
GeodiseLab (Matlab toolbox)
WSRF::Lite integration
OGSA-DAI (Database service)
WSeSS (Using SSH to tunnel requests to resources)
22
OMII_1 release:
A basic File-Compute Grid
Enables a generic computational task
Move input data from the client to the service
provider
Process the data using an application on the
service provider
Retrieve the output data from the service
provider
23
OMII Server Infrastructure
TestService
ExampleService
Allocation
Account
Data
Job
Resource Acct Happy
Mgmt Mgmt Axis
Servlet Servlet PBAC
WS-Security
AXIS Static Webpage
TOMCAT
24
Try out the OMII_1 client !
Register at www.omii.ac.uk & login
Goto the downloads page
Download the client distribution
SuSE 9.0
Client may work on other Linuxs but no exhaustive testing
Windows XP (SP 1 & 2)
Distribution requires JDK 1.4.2_04
Does not work with „just‟ a JRE
Will not work with JDK 1.4.2_05/06 & JDK 1.5.0
No testing with earlier JDKs.
25
Outline
• Middleware currently being prepared for
deployment
– Resource broker
– (NGS portal – yesterday!)
• Under assessment (by ETF for NGS):
– gLite middleware from EGEE
– OMII
– GT4
26
A Service-Oriented
Infrastructure
Users
Composition
• Service-oriented applications
– Wrap applications as Workflows
services
Invocation
– Compose applications
into workflows Appln Appln
• Service-oriented Service Service
infrastructure Provisioning
– Provision physical
resources to support
application workloads
•Carl Kesselman at Globus Week, NeSC, 4th April 27
– 8th April 2005
Summary
• Middleware currently being prepared for deployment
– Resource broker
– (NGS portal – yesterday!)
• The context for the next generation of middleware:
service orientation, based on WS and the emerging
standards
• Under assessment by the Engineering Task Force:
– gLite middleware from EGEE
– OMII
– GT4
28
Related docs
