Introduction to grids
Introduction to grids Taavi Hupponen, CSC Definition? There are as many definitions as there are grids… Power grid analogy really isn’t a very good one Grids aim to provide easy, efficient and secure access to distributed resources How to recognize a grid? • Resource sharing (cpu, storage…) • Spans over organization borders • Security • Based on open standards Grid types Categorization of grids is more or less artificial, most grids fall into several categories Computational grids • The traditional grid • Connecting clusters, workstations and supercomputers • Examples: EGEE, DEISA, SETI@home Data grids • Easy, efficient and powerful access for data • Uniform interface, distribution and replication of large data sets • Examples: Bridges, BIRN, peer-to-peer file sharing networks like BitTorrent? Knowledge grids, services grids Building blocks Most of the grids are built of same basic blocks, including • Computing elements • Storage elements • User interface • Job management • User management • Security Middleware The building blocks are implemented by the middleware of the grid Middleware acts between an application and the operating systems of the grid nodes The term ’middleware’ is used quite loosely, it can mean almost anything Examples: • LCG-2 and gLite (EGEE) • Nordugrid ARC (SweGrid, M-grid) • Unicore (DEISA) • Globus Toolkit Unfortunately middlewares don’t work very well together, work is being done to improve grid interoperability Common grid user interfaces Command-line interfaces • Still the most common way of using grids • Almost like using a batch job system in a local cluster: Write the job description Submit the job Poll for status Get the results • In addition: certificate handling Graphical clients • Often include workflow features Web portals • Either hide or expose the grid middleware • One portal for one or more grids (P-GRADE) Security in grids With most grids, security has been considered from the beginning, unlike with for example World Wide Web Grid security: • Is based on Public Key Infrastructure (PKI), which is a robust security mechanism used by for example ssh and ssl • Usernames and passwords are replaced by certificates • Certificates are provided by trusted entities called Certificate Authorities • PKI provides authentication, integrity and confidentiality Virtual organisations Access to grid resources is often controlled in Virtual Organisation level instead of individual users so VOs are based on collaboration, geographical location, scientific field Example: Biomed VO in EGEE In its simplest form: list of user identities, can also include the programs that are to be used Putting programs into the grid Programs installed by grid admins • Either at all or only at some nodes • There usually is a a common set of programs that can be found on each node of a grid (basic utilities, compilers etc.) • Nodes have mechanisms for advertising which programs are installed Programs installed by grid users • Program is sent to the node with the job description and input data • You need to consider hardware architecture, operating system and library issues What kind of problems fit into grids? Non-parallel problems • As if running on local workstation Embarassingly parallel problems • The problem is easily split into smaller independent jobs that can be distributed inside a site or even among several sites • Very well suited for grids Most problems are in-between and are best executed inside one site Grid examples EGEE • Grid of heterogenous clusters and workstations • Over 30,000 cpu, 5 Petabytes of storage • EGEE project ended in March 2006, EGEE II started in April 2006 • Funded by EU FP6 • http://www.eu-egee.org DEISA • Grid of supercomputers (mostly IBM) • For High Performance Computing applications • Funded by EU FP6 • http://www.deisa.org Challenges Constant development makes it challenging for users and admins to keep up Distribution adds overhead, decreases control and transparency Usability issues Grid interoperability issues Benefits Grids don’t increase resources – they make usage of existing resources more efficient • Load-balancing, idle resources to use Handling of large computations or data sets that aren’t possible within single site (CERN LHC) Increased collaboration Grids are still developing, but already offer good opportunities.