CMS and LHC Software and Computing Including the Caltech Tier2 and

Document Sample
CMS and LHC Software and Computing Including the Caltech Tier2 and Powered By Docstoc
					  CMS and LHC Software and Computing
The Caltech Tier2 and Grid Enabled Analysis

         Tier2   MonALISA          GAE

                     DoE Review
                    Julian Bunn
        California Institute of Technology
                   July 22, 2004
 CMS Computing & Software: Data Grid /
 Grid Projects
 Grid Analysis Environment (GAE)
 Tier2
 Distributed Physics Production
 Calorimetry/Muon Software
 Conclusion
CMS Computing and
 Data Grid model
 Data Challenges
                           LHC Data Grid Hierarchy:
                            Developed at Caltech
                                                                     CERN/Outside Resource Ratio ~1:2
                           ~PByte/sec                                Tier0/( Tier1)/( Tier2) ~1:1:1

                                            Online System              ~100-1500
Experiment                                                             MBytes/sec
                                                                                   CERN Center
                                                  Tier 0 +1                        PBs of Disk;
                                                                                   Tape Robot
Tier 1          ~10 Gbps
      IN2P3 Center                  RAL Center                       INFN Center             FNAL Center

                                                                                                             10 Gbps
                                        Tier 2                                           Tier2   Tier2  Tier2
                                                                     Tier2 CenterTier2 Center Center Center Center
             ~2.5-10 Gbps
   Tier 3
                      Institute Institute    Institute   Institute
                                                                               Tens of Petabytes by 2007-8.
Physics data cache
                                            0.1 to 10 Gbps                        An Exabyte by ~2015.
            Workstations                         Tier 4

Emerging Vision: A Richly Structured, Global Dynamic System
                            CPT Project
        CCS                          PRS                      TriDAS
Core Computing & Software   Physics Reconstruction and       Online Software
Computing Centres
                               Tracker / b-tau
CMS Computing Services
                              E-gamma / ECAL
 Architecture,                                           Online (DAQ) Software
  Frameworks / Toolkits                                  Framework
                             11. HCAL Jets, MEt
 Software Users and                                      Online Farms
 Developers Environmnt
Production Processing
 & Data Management
                            Physics Groups

                              SUSY & Beyond SM
                                Standard Model
                                 Heavy Ions
   Caltech activity
                              SCB Chair 1996-2001;
   USCMS Leadership           CCS/CPT Management Board from 2002-2003
                      DC04 Data Challenge
                                                               30 Million T0 events processed
 Concentrated on the “Organized,Collaboration-
Managed” Aspects of Data Flow and Access

                   T0 at CERN in DC04
                   25 Hz input event rate (Peak)
                   Reconstruct quasi-realtime
      FNAL         Events filtered into streams
     Chicago       Record raw data and DST
                   Distribute raw data and DST to T1’s             T1
                                                            RAL                              FZK
T1                T1 centres in DC04                       Oxford             T0           Karlsruhe
                  Pull data from T0 to T1 and store                     T1          T1
                  Make data available to PRS                                               CNAF
       Tier2 @    Demonstrate quasi-realtime “fake”    PIC                                Bologna
T2                                                   Barcelona      T1
      Caltech,     analysis of DST’s                                               IN2P3
      UFlorida,                                                                     Lyon
                     T2 centres in DC04
                     Pre-challenge production at > 30 sites
                     Modest tests of DST analysis
   CMS Computing and Core Software
          (CCS) Progress
 DC04 (5% complexity): Challenge is complete but
  post-mortem write-ups are still in progress
    Demonstrated that the system can work
      for well controlled data flow and analysis,
      and a few expert users
     Next challenge is to make this useable by
      average physicists and demonstrate that
      the performance scales acceptably
 CCS Technical Design Report (TDR): Aligned with
  LCG TDR submission (July 2005)
 DC05 (10%): Challenge Autumn 2005 to avoid
  “destructive interference” with Physics TDR
CMS Computing and
  Grid Projects
                              Grid Projects
PPDG (PI) /SciDAC                              Trillium
 Particle Physics Data Grid. Funded by             Coordinates PPDG, GriPhyN and iVDGL.
 DOE in 1999. New funding in 2004-6.               DoE and NSF working together: DOE
 Deployment of Grid computing in
                                                   (labs), NSF (universities). Strengthening
 existing HEP experiments. Mainly
 physicists.                                       outreach efforts
GriPhyN/iVDGL (Co-PI)
                                        Initially Caltech, Argonne, NCSA, SDSC,
 Grid Physics Network, international
 Virtual Data Grid Laboratory. Funded   now expanded. Massive Grid resources.
 by NSF in 1999. Grid Middleware (VDT, CHEPREO
 “Virtual Data”), Tier2 deployment and
 Grid Operations. HENP, Astronomy,      Extending Grids to South America FIU,
 Gravity Wave physics.                  Caltech CMS, Brazil
Open Science Grid
 Caltech/FNAL/Brookhaven …Combine               UltraLight
 computing resources at several DOE                Next Generation Grid and Hybrid Optical
 labs and at dozens of universities to             Network for Data Intensive Research
 effectively become a single national
 computing infrastructure for science,
 the Open Science Grid.
                Others: EU DataGrid, CrossGrid, LHC Computing Grid (LCG), etc.
                UltraLight Collaboration:
                                                       Caltech, UF, FIU,
                                                         UMich, SLAC,FNAL,
                                                         CERN, UERJ(Rio),
                                                         NLR, CENIC,UCAID,
                                                         Translight, UKLight,
                                                         Netherlight, UvA,
                                                         UCLondon, KEK,
                                                       Cisco, Level(3)

 Integrated hybrid (packet-switched + dynamic optical paths) experimental
  network, leveraging Transatlantic R&D network partnerships;
    10 GbE across US and the Atlantic: NLR, DataTAG, TransLight,
     NetherLight, UKLight, etc.; Extensions to Japan, Taiwan, Brazil
 End-to-end monitoring; Realtime tracking and optimization; Dynamic
  bandwidth provisioning
 Agent-based services spanning all layers of the system,
  from the optical cross-connects to the applications.
The Grid Analysis Environment
“Where the Physics Gets Done”
              Grid Analysis Environment

 The “Acid Test” for Grids; crucial for LHC experiments
    Large, Diverse, Distributed Community of users
    Support for 100s to 1000s of analysis tasks,
     shared among dozens of sites
    Widely varying task requirements and priorities
    Need for Priority Schemes, robust authentication and Security

 Operates in a resource-limited and policy-constrained
  global system
    Dominated by collaboration policy and strategy
     (resource usage and priorities)
    Requires real-time monitoring; task and workflow
     tracking; decisions often based on a Global system view
                                                GAE Architecture
           Analysis                 Client              Analysis                  Analysis Clients talk
            Client                                       Client
                                                                                   standard protocols to the
                         HTTP, SOAP, XML-RPC                                       “Grid Services Web
                                                                                   Server”, a.k.a. the Clarens
                                   Grid Services                                   data/services portal.
                                   Web Server
                                                                                  Simple Web service API
                                                                                   allows Analysis Clients
         Fully-                                                                    (simple or complex) to
        Planner                      Metadata                                      operate in this
                                      Virtual                                     Typical clients: ROOT, Web
                                       Data                                        Browser, IGUANA, COJAC
                      Management                   Monitoring      Applications   The Clarens portal hides
        Concrete                     Replica                                       the complexity of the Grid
                                                                                  Key features: Global
                                                                                   Scheduler, Catalogs,
                                                                                   Monitoring, and Grid-wide
                                    Execution                                      Execution service.
Support for                         Manager
the physics                         Grid Wide
analysis and                        Execution
computing model                      Service
Structured Peer-to-Peer GAE

             The GAE, based on
               Clarens and Web services,
               easily allows a “Peer-to-
               Peer” configuration
               to be built, with associated
               robustness and scalability
             The P2P configuration
               lows easy creation, use
               and management of
               complex VO structures.
      Grid-Enabled Analysis Prototypes
     Collaboration                     COJAC
    Analysis Desktop             (via Web Services)

ROOT (via Clarens)

                                   (via Clarens)
     GAE Integration with CMS and LHC
 Clarens Servers: Python and Java versions available
 Refdb: (Stores Job/Task parameters or “cards”) Replica of DC04
        production details available on Tier2
 POOL: (Persistency framework for LHC) 60 GB POOL file catalog has
        been created on Tier2, based on DC04 files.
 MCRunjob/MOP: (CMS Job/Task workflow description for batch):
        integration into the Clarens framework, by FNAL/Caltech
 BOSS: (CMS Job/Task book-keeping system) INFN is working on the
        development of a web service interface to BOSS
 SPHINX: Distributed scheduler developed at UFL
 Clarens/MonALISA Integration: Facilitating user-level Job/Task
        monitoring: Caltech MURF Summer Student Paul Acciavatti
 CAVES: Analysis code-sharing environment developed at UFL
 Core System: Service Auto Discovery, Proxy, Authentication..

R&D of middleware grid services for a distributed data analysis system:
   Clarens integrates across CMS/LCG/EGEE and US Grid Software
                    GAE Deployment
 20 known Clarens deployments Caltech, Florida (9 machines),
  Fermilab (3), CERN (3), Pakistan (2+2), INFN (1)
 Installation of CMS (ORCA, COBRA, IGUANA,…) and LCG
  (POOL, SEAL,…) software on Caltech GAE testbed for
  integration studies.
 Work with CERN to include the GAE components in the CMS
  software stack
 GAE components being integrated in the US-CMS DPE
 Demonstrated distributed multi user GAE prototype at SC03
  and elsewhere
 Ultimate goal: GAE backbone (Clarens) deployed at all Tier-N
  facilities. Rich variety of Clarens web servers offering GAE
  services interfaced with CMS and LCG software
    CMS core software and grid middleware expertise
                   GAE Summary
 Clarens Services-Fabric and “Portal to the Grid”
  Numerous servers and clients deployed in CMS
  Integration of GAE with MonALISA progressing:
  A scalable multi-user system

 Joint GAE collaborations with UFlorida, FNAL and
 PPDG “CS11” very productive
  Production work with FNAL
  Mentoring PostDocs, CS Students, Undergraduates

 Rationalising the new EGEE ARDA/gLite work with GAE

 GAE project description and detailed information:
History and Current Status
         Caltech Tier2 Background
 The Tier2 Concept Originated at Caltech in Spring 1999
 The first Tier2 Prototype was proposed by Caltech
   together with UCSD in 2000.
   It was designed, commissioned and brought into
    production in 2001.

 It Quickly Became a Focal Point Supporting A Variety
  of Physics Studies and R&D Activities
   The Proof of concept of the Tier2 and The LHC
    Distributed Computing Model
   Service to US CMS and CMS for Analysis+Grid R&D
   Production: CMS Data Challenges, Annually from Fall 2000;
    H and  Calibration Studies with CACR, NCSA, etc.
   Grid Testbeds: Development  Integration  Production
   Cluster Hardware & Network Development
                         Tier2 – Example Use in 2001
                                     Full Event                                                                                          Full Event
                                     Database of                                                                                        Database of
                                    ~40,000 large                                                                                      ~100,000 large
                                       objects                                                                                            objects


                          Parallel tuned GSI FTP                     “Tag”                                Parallel tuned GSI FTP
                                                                   database of
                                                                  small objects

              Bandwidth Greedy Grid-enabled Object Collection Analysis
                               for Particle Physics
                   Julian Bunn, Ian Fisk, Koen Holtman, Harvey Newman, James Patton
                   The object of this demo is to show grid-supported interactive physics analysis on a set of 144,000 physics events.
  Initially we start out with 144,000 small Tag objects, one for each event, on the Denver client machine. We also have 144,000 LARGE objects,
                                             containing full event data, divided over the two tier2 servers.
Using local Tag event database, user plots event parameters of interest
User selects subset of events to be fetched for further analysis
Lists of matching events sent to Caltech and San Diego
Tier2 servers begin sorting through databases extracting required events
For each required event, a new large virtual object is materialized in the server-side cache, that contains all tracks in the event.
The database files containing the new objects are sent to client using Globus FTP; client adds them to local cache of large objects
The user can now plot event parameters not available in the Tag
Future requests take advantage of previously cached large objects in the client
                               Current Tier2
              Newisys Quad                                     Caltech-    Caltech-
             Opteron Network                                    Grid3        DGT


                                                                                        APC UPS
                                                                                        Dell 5224

Cisco 7606
 Switch/                                        Network
  Router                                       Managem-                      Network &
                                               ent Server                 MonALISA Servers

                               Quad Itanium2                          tier2c
                               Windows 2003
            2003 Bandwidth Challenge “Data
             Intensive Distributed Analysis”
SuperComputing 2003
 Multiple files of ~800k simulated
  CMS events, stored on Clarens
  servers at CENIC POP in LA, and
  TeraGrid node at Caltech.
 Transferred > 200 files at rates up
  to 400MB/s to 2 disk servers at
 Convert files into ROOT format,
   then publish via Clarens
 Data Analysis results displayed
  by Clarens ROOT client

Sustained Rate: 26.2 Gbps (10.0 + 8.2 + 8.0)

Note: This Bandwidth Challenge subsequently prompted the gift to us of a 10G
wave to Caltech Campus (and thence the Tier2).
              Caltech Tier2 Users Today
Institute             Users    The Tier2 is supporting scientists
Caltech               34        in California and other remote
CERN                  8         institutes
UC Davis              6        Physics studies
                                (Caltech, Davis, Riverside, UCLA)
Kyungpook Natl.       5
Univ., Korea                   Physics productions
Univ. Politechnica,   4         (Caltech, CERN, FNAL, UFlorida)
Bucharest                      Network developments and
Univ. Florida         3         measurements
FNAL                  2         (Caltech, CERN, Korea, SLAC)
UC San Diego          2        MonALISA (Caltech, Romania)
Univ. Wisconsin       2        CAIGEE collaboration
SLAC                  2         (Caltech, Riverside, UCSD, Davis)
UC Riverside          1        GAE work
UCLA                  1         (Caltech, Florida, CERN)
TOTAL                 70
   US-CMS ASCB “Tier2 Retreat”
                           Two day “Tier2 Retreat” hosted
                            by Caltech
                             To Review Progress at the 3
                               Proto-Tier2s; to Update the
                               Requirements, and the Concept
                           The start of the procedure by
                            which the production US-CMS
                            Tier2 Centres will be chosen
                           Outcomes:
                             Fruitful discussions on Tier2
                              operations, scaling, and role in the
                              LHC Distributed Computing Model
                             Existing prototypes will become
                              Production Tier2 centres;
                              recognising their excellent progress
                              and successes
                             Call for Proposal document being
                              prepared by FNAL: Iowa, Wisconsin,
                              MIT, expected to bid
 Physics Channel Studies:
  Monte Carlo Production
CMS Calorimetry and Muon
    Endcap Software
              CMS Events Produced by Caltech
                for Higgs and Other Analysis
 Themes: H and
  Gravitons, 
  Calibration; Events for       Cluster     2000 2001    2002   2003    2004
  DC04 and Physics TDR.                                                [6 Mo.]
                                Generated    3M   4.2B   5.2B   16.7 B 37 B
 Tier2 & Grid3
 Caltech: We have largest
                                Simulated    3M   3.3M   2.8M   7.2M   11.2M
  NRAC award on TeraGrid
 NCSA Clusters: IA64 Tera-
                                Recon-       3M   3.3M   2.8M   7.2M      (***)
  Grid; Platinum IA32; Xeon
 SDSC: IA64 TeraGrid cluster
 NERSC: AIX Supercomputer
 UNM: Los Lobos cluster        Analyzed     At  2.3M    2.3M   10M       (***)
*Madison: Condor Flock                     CERN                (**)

 PYTHIA, CMSIM,                     Total 2.6 M Node Hours of CPU Time
  CERNLIB,                      (**) Some datasets analyzed more than once
  GEANT3 – ported to IA64.      (***) To Be Done in Second Half of 2004
              Calorimetry Core Software
                    (R.Wilkinson, V.Litvin)
 Major “Skeleton Transplant” underway:
   Adopting the same “Common Detector”
    core framework as the Tracker & Muon
   Use of common detector framework
    facilitates tracking across subdetectors
 Common Detector Framework also allows
   Realistic grouping of DAQ readout
   Cell-by-cell simulation on demand
   Taking account of Misalignments
 Developers:
   R.Wilkinson (Caltech) : Lead design +
   V. Litvin (Caltech) preshower
   A. Holzner (Zurich-ETH) :
    geometry + ECAL barrel
   H. Neal (Yale): ECAL endcap
              Endcap Muon Slice Test DAQ
 Software developed by R. Wilkinson in
  2003-2004 now adopted by the chamber
   Handles VME configuration
     and control for five kinds of boards
   Builds events from multiple
     data & trigger readout PCs
   Supports real-time data
   Unpacks data
   Packs simulated data into
      raw format
 Ran successfully at the
  May/June 2004 Test Beam
 Now supported by Rice U.
                                    Builds on Expertise in EMU Software,
  (Padley, Tumanov, Geurts)
                                       Reconstruction and Trigger; and
 Challenges ahead for Sept.
  EMU/RPC/HCAL tests                     Large Body of Work in 2001-3
              Summary & Conclusions (1)
 Caltech Originated & Led CMS and US CMS Software & Computing
  in the Early Stages (1996-2001)
   Carries Over Into Diverse Lead-Development Roles Today
   Leverages strong Grid-related support from DOE/MICS and NSF/CISE
 The Tiered “Data Grid Hierarchy” Originated at Caltech is the
  Basis of LHC Computing
   Tier 0/1/2 roles, interactions and task/data movements increasingly
     well-understood, through data challenges on increasing scales
   Caltech’s Tier2 is now a well-developed multi-faceted production and R&D
     facility for US CMS physicists and computer scientists collaborating on
     LHC physics analysis, Grid projects, and networking advances
   Grid projects (PPDG, GriPhyN, iVDGL, Grid3, UltraLight etc.) are
     successfully providing infrastructure, expertise, and collaborative effort
CMS Core Computing and Software (CCS), Physics Reconstruction and
   Selection (PRS) Groups: Excellent Progress
   Major confidence boost with DC04 – managed productions peaking at
     rates > 25Hz; Caltech Tier2 and MonALISA Took Part
             Summary & Conclusions (2)
Caltech leading the Grid Analysis Environment work for CMS
   Clarens Web-services fabric and “Portal to the Grid” Maturing
   Joint work with UFlorida, CERN, FNAL, PPDG/CS11 on system
    architecture, tools, and productions
   Numerous demonstrators and hardened GAE tools and clients
    implemented, and serving US CMS and CMS Needs
   Integration with MonALISA to flesh out the Distributed System Services
    Architecture need for a managed Grid on a global scale
Groundbreaking work on Core CMS Software and Simulation Studies
   HCAL/ECAL/Preshower common software foundation, and
    EMU DAQ software for CMS Slice Test: developments led by Caltech
   Higgs channel background simulations at very large scale, leveraging
    global resources opportunistically
   Enables US CMS Lead Role in H  Analysis: Caltech + UCSD
                 GAE Recent Prototypes

Clarens BOSS Interface            Clarens Virtual Organization

Clarens POOL Interface         Clarens Remote File Access
        Event Generation, Simulation and
     Reconstruction for CMS – Caltech Tasks

 Cluster   2000 2001 2002 2003                         2004
UW-Madison 204 k 124 k --  262 k                       319 k
Platinum           n/a      0.6 k         --   515 k   434 k
Los                 --      33 k          3k    --       --
Tungsten           n/a       n/a      n/a       n/a    279 k
Tier2 CIT         n/a       50k     200k 300k           25k
TG Total          n/a       n/a      n/a      n/a      199 k
 Total Node      204 k 208 k 203k once
(**) Some datasets were reanalyzed more than 1077 k    969k
(***) To Be Done in Second Half of 2004
                                                       [6 Mo.]
           Grid Enabled Analysis: User View
              of a Collaborative Desktop
 Physics analysis requires varying levels of interactivity,
    from “instantaneous response” to “background” to “batch mode”
   Requires adapting the classical Grid “batch-oriented” view to a
    services-oriented view, with tasks monitored and tracked
   Use Web Services, leveraging wide availability of commodity tools
     and protocols: adaptable to a variety of platforms
   Implement the Clarens Web Services layer as mediator between
    authenticated clients and services as part of CAIGEE architecture
   Clarens presents a consistent analysis environment to users, based on
    WSDL/SOAP or XML RPCs, with PKI-based authentication for Security

      External Services

     Storage Resource Broker


                               Iguana           Browser                    MonaLisa
       Cluster Schedulers                                                                           ROOT
         ATLAS DIAL

         Griphyn VDT
                               VO Management                    Clarens               File Access
     MonaLisa Monitoring
                               Authentication   Authorization         Logging     Key Escrow         Shell