SEAL Project Overview
ACAT03 1-5 December, 2003
KEK
IX International Workshop on Advanced Computing
and Analysis Techniques in Physics Research
Lorenzo Moneta/ CERN-EP
on behalf of the SEAL team
Outline
Introduction
– Scope and motivation
SEAL Work Packages
– Description
– Current status
Future plans and summary
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 2
Project motivation
What is SEAL ?
– project in LCG Application Area
– started last year after BluePrint RTAG recommendations
– common project on core tools and services
» SEAL: Shared Environment for Applications at LHC
SEAL aims to
– Provide the software infrastructure, basic frameworks, libraries
and tools that are common among the LHC experiments
– Select, integrate, develop and support foundation and utility class
libraries
– Develop a coherent set of basic framework services to facilitate
the integration of LCG and non - LCG software
Do not re-invent the wheel
– Try to use as much as possible of existing software
– Develop/adapt/generalize to achieve necessary level of coherency
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 3
Project scope
Foundation Class Libraries
Software Structure
– Basic types (STL, Boost,…), utility
libraries, system isolation libraries,
Applications domain specific foundation libraries
Mathematical Library
Reconstruction
Visualization
Framework
Framework
Framework
Simulation
Frameworks
– Basic mathematical functions, linear
... algebra, fitting and minimization
Other
libraries
Basic Framework Services
Basic Framework – Component model, reflection, plug-in
management, incident (event)
Foundation
Libraries
Optional Libraries management, distributed computing,
grid services, scripting
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 4
Project Work Packages
Foundation Foundation and Utility Libraries and Plug-in Manager
MathLibs Math Libraries Support and Coordination
Dictionary LCG Object Dictionary
Framework Component Model and Basic Framework services
Scripting Scripting Services
Grid Grid Services Not yet active
Documentation Education and Documentation
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 5
Foundation and Utility Libraries
Inventory of existing C++ utilities (http://seal.cern.ch/components.html)
– Recommended classes useful for LCG projects and experiments
– Large number from Boost (open source project)
» Support for Boost is provided (in contact with developers)
Develop in SEAL what is missing in STL and Boost
Have a set of low-level, fairly independent class libraries
– 200 classes originated mainly from classlib (CMS)
– Basic abstraction and foundation utilities classes
– Timing utilities
– Stream-oriented I/O, compression and decompression
High quality, efficient, well tested and documented library
– Recommend best practice to LHC developers
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 6
Plug-in Manager
Plug-in Management
– Service in charge of managing, querying, [un]loading plug-ins
Low level plug-in management (no framework constraints)
– Ability to create ―module‖ libraries containing ―plug-ins‖
– Ability to instantiate concrete implementations knowing the plug-in
category and the concrete type
– Caching information for fast loading
» what modules contain
» what kind of plug-ins exist and which categories
Imported from CMS IGuana
– Basic concept: advanced object factory
» Interfaces for object instantiation and plug-in provider
– Dynamic loading completely orthogonal — and optional!
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 7
Mathematical Libraries
GSL (Gnu Scientific Library) evaluation
– Extensive comparison with NagC
– Validation and test suite
Re-implementation of MINUIT in C++
– Prototype available with Migrad and Minos
– Undergoing numerical and performance tests
» Comparison tests with Fortran version (compatibility within errors)
Linear Algebra
– Prototype wrapper around GSL matrices.
– Comparison of various linear algebra packages (CLHEP, GSL, [u]CBLAS)
Participation in CLHEP project
Provide to experiments math and statistics libraries to be
used in analysis, reconstruction and simulation.
– support, consultancy, etc..
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 8
Object Dictionary
Adding ―introspection/reflection‖ capabilities to C++
Use for persistency and interactivity (e.g. Python)
Modern languages provide reflection inherently
– In C++ reflection is very limited (RTTI)
The basic C++ concepts shall be supported
– e.g. inheritance, methods, data members, accessibility, templates
Seal provides :
– Reflection Builder and Reflection (API interface) packages
– Dictionary generator to produce dict. source from C++ files
» Uses gcc-xml + Python script
– Dictionary for some standard packages (STL, CLHEP)
Used by POOL (persistency)
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 9
Dictionary Generator
gcc-xml to parse header-files (extension to gcc )
XML file generated with dictionary information
Python script to generate dictionary C++ code
– use an additional XML file to customize class selection
Compiled and loaded at run-time to create dictionary
– Provide meta information about the object to the user
User
.xml selection file
.xml par filter gendict
gccxml
.h ser +
exten
_dict.cpp make ..Dict.so
(Python script) lcgdict
.h
.h Reflection
.h Clients
#include files Libraries
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 10
Framework Services
Motivation:
– Allow LCG developers and users to write portable and modular code
– Provide abstractions for modelling software systems
Component model definition
– Basic Components
» Set of ―base classes‖ and interfaces needed to build more
complex components
» Provide standard functionality
instantiation, configuration, lifetime management, etc
Users (framework developers) should model their services
using these proposed base classes
SEAL provide in addition a set of basic services
implemented following the component model
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 11
SEAL Component Model
Context
Application
C1 C1
C2 C1
C5
Svc1 Svc2
C1
C3 C1
C4 C1
C6 C1
C7
Svc1
Component
C1
C7
Service
Hierarchy of bases classes
– Components living in a Context
– Services are Components providing their local Context
User classes inherit from Component or Services
– Plug-in functionality for free
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 12
Basic Common Framework Services
Application Service
– Defines the top level Context
– Defines initial set of Components to be loaded in the application
Message Service
– Message composition, filtering and reporting
Configuration Service
– Management of Component properties and loading configurations
– Multiple backends foreseen:
» Gaudi style options, .INI style, CMS style, XML, ...
Other services will come later:
– Object Whiteboard (Transient Object repository)
– Dictionary Service (Loading Dictionaries in the application)
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 13
Scripting Services
Scripting needed for Interactivity, Configuration, Rapid
prototyping, Integration (Component bus), etc.
Main scripting language :
– Python EDG API PVSS XML Database
GUI
GUI
Define guidelines for Python
developing Python JPE PyROOT PyGaudi math
shell
math
bindings :
Gaudi
– Various tools available Java Root
Frame
(Boost.Python, SWIG) Classes Classes
work
– Recommend solutions:
» Boost.Python
» LCG Dictionary
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 14
Scripting Packages
PyLCGDict
– Python extension module to the LCG dictionary
– Provide access to C++ libraries from Python, ―for free‖
– Automatically generates Python proxies for C++ objects
using dictionary information
» Everyone can create binding effortlessly
PyROOT
– Python bindings for ROOT classes
» access to ROOT functionality from Python
– Binding using the ROOT dictionary (CINT)
– Missing gateway ROOT -> Python
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 15
Education/Documentation
Produce documentation
– Reference documentation (generated with doxygen)
– HowTo pages for the released elements
» Plugin Manager, Dictionary generation, PyROOT, PyLCGDict, etc.
– Release notes
Training material (tutorials)
– Help developers incorporating SEAL into the LCG projects and the
experiment frameworks
Python courses
– Hands-on introduction course to Python programming
– Very positive feedback received
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 16
Software Process and QA
Design
– Team design sessions
– Prototypes using Pythons
Code Review
– No formal code reviews
– Coding done very often in pairs (XP style)
– More than one developer knowledgeable for each package
Testing
– More than 200 SEAL unit tests based on CppUnit
– All are driven and run automatically by QmTest
Bug reporting and tracking
– Savannah Portal (http://savannah.cern.ch/projects/seal)
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 17
Future plans (1)
Foundation libraries (Plug-in Manager)
– Follow Boost developments
– Little development planned in SEAL (hash-maps)
MathLibs:
– Produce inventory of mathematical functions
– Evolve Minuit C++ into a final product
– Linear Algebra studies
– Generate dictionary library for Python and CINT
Dictionary:
– Implementation of new reflection model
» Overcome some existing limitations (typedef,…)
– Converge to a common dictionary with CINT ?
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 18
Future plans (2)
Framework:
– Develop Whiteboard (object repository) and Dictionary service
(loading of dictionary libraries on-demand)
Scripting :
– PYLCGDict:
» Migrate much of functional core from C++ to Python
» Support more natural Python features (eg iterator protocol)
– PyROOT:
» Unify core with PyLCGDict
» Develop gateway ROOT -> Python
Grid :
– New LCG project (ARDA) is starting
– Expect need to interface to some of the ARDA services
Documentation:
– Produce tutorials for foundation classes, Boost, etc..
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 19
Summary
SEAL has delivered basic foundation, utility libraries and
object dictionary
– The main ―client‖ have been POOL (persistency project) and PI
» Plug-in manager is also used in CMS
– Very positive feedback received for plug-in manager
– Currently being integrated into experiments’ frameworks of LHCB
(Gaudi) and ATLAS (ATHENA)
The first version of the Component Model and Framework
services is available
– Start integrating in POOL and the LHC experiments
– Wait for feedback before developing more services
Scripting based on Python
– Boost.Python and PyLCGDict recommended for Python bindings
– PyLCGDict is used to provide bindings for classes of Physics
Interfaces (PI) project
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 20
Conclusions
http://seal.cern.ch/
Acknowledgments:
Seal team members:
R. Chytracek, J. Generowicz, W.
Lavrijsen, M. Marino, P. Mato,
S. Roiser, L. Tuura, M. Winkler
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 21
Additional Slides
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 22
LCG Application Domain Coverage
Fitter
Algorithms Scripting
EvtGen Engine NTuple
Interactive
GUI
Reconstruction Analysis Services
Event Detector
Generation Simulation
Modeler
Geometry Event Model Calibration
FileCatalog
StoreMgr Scheduler
Dictionary PluginMgr
Whiteboard Monitor Grid
Persistency Core Services Services
Foundation and Utility Libraries
ROOT GEANT4 FLUKA MySQL DataGrid Python Qt ...
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 23
Plug-in Manager Design
Originated from CMS Iguana
PluginManager PluginCache
*
PluginFactoryBase ModuleBase
*
T
PluginFactory PluginInfo Module ModuleDef
*
XXXFactory XXXInfo MyModuleDef
Module Library
Client
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 24
Reflection System
Reflection Builder: the loading interface
Reflection : the user API
NamespaceBuilder ItemBuilder
ClassBuilder FieldBuilder MethodBuilder
ReflectionBuilder
Item PropertyList
Namespace Class Field Method
Reflection
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 25
LCG Object Dictionary: Usage
.adl
.h
.xml
Population
ROOTCINT GCC -XML ADL/GOD
CINT generated
Dict generating
code
code
Conversion (1) in
Dict gateway
LCG to CINT LCG
CINT Dictionary Other
Streamer
Dict Clients:
(python,
ROOT I/O (2) out GUI, etc.)
Reflection
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 26
Seal Component Model
Hierarchy of bases classes
Each Component lives in a Context
Contexts form their own hierarchy Component
Factory
Support for component look-up
Component
A Service provides its own local Loader
Context Component *
– Components of a Service live in the
scope defined by its own Context 1
User classes inherit from Service
1
Context
Component or Service *
– Plug-in functionality
for free MyComponent MyService
ACAT03, 1-5 Dec. 2003 SEAL Project Overview L. Moneta / CERN 27