Grid Packaging Technology
Technical Talk
University of Wisconsin Condor/GPT
Meeting
January 3,4 2002
Michael Bletzinger
mbletzin@ncsa.uiuc.edu
National
Specialist, Software Developer
Computational
Environment
Advanced ComputationalScience
and Security, NCSA
National Center for Supercomputing Applications National Computational Science
GPT Purposes
" Give developers an easy way to package software
– All packaging data is specified from a source perspective.
– Packaging data is strongly typed.
" Provides a means of building and installing
collections of packages
" Provides standard version negotiation method for
managing updates to packages.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
GPT and Globus
" GPT was designed to re-implement the
deployment requirements of Globus plus shared
libraries.
– Globus most of Globus is libraries.
– Globus has deployment requirements for client tools,
service daemons, and development/build environments.
– Globus wants to release individual components
separately
National
– Globus wants to add outside software (ie. openssl,
Computational
openldap) to these deployments. Science
– Globus wants to work on multiple platforms.
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Splitting up the Source Code
" Source Package Types
– Source packages: contain source code and
documentation.
– Setup packages: contain configuration/localization scripts
and files.
– Test Packages: contain component test code.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Source Packages
" Contains source code for programs, scripts, and
libraries.
– All libraries are added to the link line.
" Contains documentation and data files.
" All of these share the same version number.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Other Packages
" Setup Packages
– Contains scripts and templates.
– These are assumed to have no build flavor.
– Generate files outside of the control of the packaging
system.
" Test Packages
– Contains test harnesses and code.
– Installed in a special directory. National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Binary Package Types
Multi-flavored
Type Description Flavored installation
pgm programs and scripts. Yes No
shared libraries and flavored
rtl scripts. Yes Yes
dev headers and static libraries. Yes Yes
doc documentation. No National
No
Computational
data non-localized data files. No No
Science
National Center for Supercomputing Applications National Computational Science
Package Transformations
Source Code
Test Source Setup
Package Package Package
pgm doc pgm doc pgm doc
data rtl dev data
National
data
Computational
Science
localized
Installed Files files
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Dependencies
" A Dependency is a need relationship to another
package
– RPM's "Requires:" does the same thing.
" Different Dependency Types for Source an
Binary Packages.
" Types include both build and runtime
dependencies.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Runtime Dependencies
" Source Runtime dependencies are transferred to
the specific binary package.
" Runtime dependencies require a binary package
type in the specification .
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Build Dependencies
" Are transferred to the binary dev packages
" Are used in the configure stage of the source
package build.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Source Runtime Dependencies
" pgm_runtime
– Dependencies for the programs.
" lib_runtime
– Dependencies for the libraries.
" doc_runtime
– Dependencies for the documentation.
" data_runtime
National
– Dependencies for the data files. Computational
Science
National Center for Supercomputing Applications National Computational Science
Source Build Dependencies
" compile
– Requires header file from another package
" pgm_link
– Programs require libraries from another package
" lib_link
– Libraries depend on libraries from another package
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Dependencies
Header
File Source Package Script
Program Source
Package Foo Code
Package Foe
Library Source
Code
Documentation
Library Doxygen
Page
Package Fee
National
Computational Fum
Package
Science
National Center for Supercomputing Applications National Computational Science
Binary Dependencies
" Compile
– Only in dev packages
" Build_Link
– Only in dev packages
" Runtime_Link
– rtl and pgm packages
" Regeneration
National
– Only in pgm packages Computational
" Runtime Science
– in any binary package
National Center for Supercomputing Applications National Computational Science
Source to Binary Dep Conversion
Source Binary
Dependencies Dependencies Pass Thru
compile compile
Dynamic Linking
runtime runtime Static Linking
build_link
lib_link
National
runtime_link
pgm_link Computational
Science
regeneration
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Build Flavors
" A Build Flavor is a set of compilation and linking
options which have to be defined to produce
binaries that can be linked with each other.
" A development environment issue for older
versions of Globus.
" Now a runtime issue with GT 2.0 because of
shared libraries and plugins.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Build Flavors
" Build Flavor Choices
– Compiler Choice: gcc, vendor cc
– Size: 32 bit, 64 bit
– Debugging: Debug Symbols included or stripped
– Thread Package: pthreads, solaris, sproc, none
– message passing library: mpi, none
" Standard Flavors in green
National
" Example Computational
– gcc32dbgpthr = gcc compiler, 32 bit, debugging symbols,
Science
pthreads
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Versioning Purposes
" Source code change identifier.
" Distribution identifier.
" Marketing tool.
" Interface compatibility identifier
– for APIs.
– for protocols.
– for configuration files.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Version Negotiation.
" Agreement on compatibility between package
developer and package user.
– Package developer makes a compatibility
recommendation.
– Package user choose to accept or override the
recommendation.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Aging Version
" Version scheme used by the package developer.
" Based on libtool and other shared library schemes.
minor version number - indicates
major version number a binary compatible change
4.1.3
National
age - indicatesComputational
Science
backward compatibility to the
major version number
National Center for Supercomputing Applications National Computational Science
Version Requirements
" Simple Version Requirement.
– Package user specifies major version number.
– Trusts the developer compatibility recommendation.
" Version Range Requirement.
– Package user specifies a range of major and minor
version numbers.
– Does not use the compatibility recommendation from the
package developer. National
" Combinations of these can be used for more
Computational
Science
complex requirements.
National Center for Supercomputing Applications National Computational Science
Versioning Example
Version for the package is 5.1.3
Version Requirement Is Compatible
3 Yes
1 No
4.2 to 5.4 Yes
4.2 to 5.0 No
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Installation Objectives
" Standard structure for both runtime and build
installations.
" Multiple build flavors can exist in the same
location.
" Packaging data co-located with the installation.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Installation Structure
programs bin
sbin
lib
libraries
libexec
gcc32
$GLOBUS_LOCATION include headers
gcc32pthr
headers
etc packaging
metadata
National
Computational
setup
Science
test
National Center for Supercomputing Applications National Computational Science
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
How to move an installation.
" Copy Directory.
" Re-run setup scripts.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Bundling Objective
" Be able to install a collection of packages or even
an entire distribution with one command.
– Packages are not installed if compatible ones already
exist.
– All setups are consolidated into one post install script.
" Be able to build a binary bundle with one
command.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Bundling Data
" Name and version of the package.
" Binary package types that are needed.
" Build options to be avoided ie. no threads.
" Build Flavors are chosen during builds for each
platform.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
Setup Objectives
" Allow installed packages to be configured with a
script.
– Each script packaged in a setup package.
" Script execution is recorded in the packaging
data.
" Setup packages can be replaced by distributing
organizations to tailor a distribution.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Setup dependencies
" Source package indicate a setup need by a special
setup dependency.
" Setup packages have two names and versions.
– One name/version for the specific package.
" different setup packages have different name/version.
– One name/version to fulfill the setup dependency.
" All setup packages will have the same name/version for this.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
Setup Package Example
Setup
Package
Foe
Regular
Setup Package
Package Foo
Fum
National
Setup
Package Computational
Fie Science
National Center for Supercomputing Applications National Computational Science
Outline
" Splitting up the source code
" Binary Package Types
" Dependencies
" Build Flavors
" Versioning
" Installations
" Bundling National
Computational
" Setup Science
" The GPT Tools
National Center for Supercomputing Applications National Computational Science
What does GPT offer?
" An easy way for developers to package their
software.
" An easy way for people to build large sets of
source packages.
" Provides a package manager for those systems
that don't have one.
" Provides packaging metadata in an XML format.
National
" Compatible with other package managers
Computational
Science
National Center for Supercomputing Applications National Computational Science
An easy way for developers to package their
software
" gpt_setup - Sets up Makefiles and autoconf to use
packaging system.
" gpt_edit - GUI to allow developers to edit the
metadata.
" patch-n-build - for those developers that want to
keep their own build system.
" globus-makefile-header - for those who want to
build applications using Globus components.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
An easy way for people to build large sets of
source packages.
" globus-build -swiss army knife for building
packages, bundles, and development directories.
globus-build [ \
-verbose \
-help \
-installdir=path_to_installation \
-builddir=path_to_build directory \
-log=logfile \
-srcfile=source_metadata_file \
-srcdir=source_directory \
-static= \
-install-only \
-force \ National
-std-flavors \ Computational
-all-flavors \
]\ Science
[macro=value] [list of flavors to build] [list of source packages/bundles]
National Center for Supercomputing Applications National Computational Science
Provides a package manager for those systems
that don't have one.
" globus-install - Installs/uninstalls binary packages
" gpt_verify - Verifies that a Globus installation is
complete.
National
Computational
Science
National Center for Supercomputing Applications National Computational Science
NMI Changes to GPT.
" Changes for April NMI release.
– New schema/dtd for packaging data.
" conversion script to update existing packages
– Add package signatures and other installation verification
mechanisms.
– Add tools for installation management.
" Centralized file which records installations and installed
bundles.
– De-globus-ize the tools National
Computational
" replace "globus" with something else for script names variables
Science
etc.
– probably keep old globus names for compatibility purposes.
National Center for Supercomputing Applications National Computational Science