Learning Center
Plans & pricing Sign in
Sign Out

CSR Charts


									     GLAST LAT Project                        ISOC CDR, 4 August 2004

Gamma-ray Large
                         GLAST Large Area Telescope:
Area Space
Telescope                Instrument Science Operations Center

                         Section 3.4
                         Science Data Products and
                         Science Analysis Software

                         Richard Dubois
                         SAS System Manager

Document: LAT-PR-04500                        Section 3.4          1
     GLAST LAT Project                     ISOC CDR, 4 August 2004

     • Introduction to SAS Scope and Requirements

     • Overall Test Plan

     • Data Challenges

     • DC1 Summary

Document: LAT-PR-04500                     Section 3.4          2
     GLAST LAT Project                             ISOC CDR, 4 August 2004

           Science Analysis Software Overview
• Data Pipelines
     – Prompt processing of Level 0 data through to Level 1 event
     – Providing near real time monitoring information to the ISOC
     – Transient sources
     – Monitoring and updating instrument calibrations
     – Reprocessing of instrument data
• Performing bulk production of Monte Carlo simulations
• Higher Level Analysis
     – Creating high level science products from Level 1
          • Point source catalogue
     – Providing access to event and photon data for higher level data
• Interfacing with other sites (sharing data and analysis tool
     – Mirror PI team site(s)
     – SSC
• Supporting Engineering Model and Calibration tests
• Supporting the collaboration for the use of the tools
Document: LAT-PR-04500                             Section 3.4          3
     GLAST LAT Project                                                            ISOC CDR, 4 August 2004

                     Level III Requirements Summary
 Ref: LAT-SS-00020
             Function                 Requirement                  Expected Performance (if          Verification
  Flight Ground Processing    perform prompt processing         keep pace with up to 10 GB       demonstration
                              from Level 0 through Level 1      Level 0 per day and deliver to
                                                                SSC within 24 hrs
                              provide near-real time            within 6 hrs                     demonstration
                              monitoring to IOC
                              maintain state and                                                 demonstration
                              performance tracking
                              facilitate monitoring and                                          demonstration
                              updating of iinstrument
                              archive all data passing          > 50 TB on disk and tape         demonstration
                              through                           backup
  Instrument Design Support   Create simulation tool, based                                      system test -
                              on instrument geometry, that                                       comparison to
                              reproduces the interactions of                                     balloon flight and
                              photons and background                                             existing data
                              Create physics model of                                            system test -
                              expected photons and                                               comparison to
                              backgrounds incident upon                                          balloon flight and
                              the instrument                                                     existing data
                              Create algorithms to interpret                                     system test -
                              the data from the instrument                                       comparison to
                              to identify the interaction and                                    engineering model
                              estimate photon direction and                                      tests
                              Create algorithms to generate                                      system test - in
                              calibration constants for the                                      conjunction with
                              subsystem components                                               engineering model
  High Level Tools            Interface with the SSC and PI                                      demonstration
                              mirror sites, sharing selected
                              data and algorithms
                              Create High-Level Science         .                                demonstration
                              products. Development of
                              analysis tools
  Mission Support             Support the Software system                                        demonstration
                              for the life of the mission

Document: LAT-PR-04500                                                           Section 3.4                 4
                   GLAST LAT Project                                                                                            ISOC CDR, 4 August 2004


                          LAT sim/recon                                            ScienceTools                                                         Infrastructure

System Architect                          Simulation                Architects                    Analysis Tools                    Code Distribution               Release Manager

                                                            ACD                                                    Likelihood        System Tests                         Data Server

                                                            CAL                                                     Pulsars          Issues Tracker                      Documentation

                                                                                                                                      SLAC Linux                     SLAC Windows
                                                            TKR                                                      GRBs
                                                                                                                                      environment                     Environment

                                                           GEANT4                                                  Obs Sim           Sundry Utilities                     Caliibrations

                                                                                                                                                                    Code Management
    Recon                           Event Interpretation          User Interface             Release Manager                         Event Display

                    ACD                                                I/O



                                                                         There is a name for each box, except Infrastructure lead.
 System Tests                        Release Manager
                                                                         Asking for SLAC to fill this position. Making do for now.
   Flight Int

       Document: LAT-PR-04500                                                                                                   Section 3.4                                 5
        GLAST LAT Project                                                                      ISOC CDR, 4 August 2004

                        SAS in the Ground System
                             Mission Operations Center

                                    Level 0 & HSK
Pipeline (Gino) is robotic

                                                    Instrument Operations Center        Contingency
                                                                (IOC)                   Commanding
backbone of ISOC process                                                                                             Nominal
handling –                                                                                    Commanding
                                 Data Processing Facility                                                                             Center
Performs L1 & L2                                    Housekeeping      Science Data
                                                                                                      Level 1 Products
processing                                              Data           Production

                                  Instrument Monitoring                Performance
                                                                     Evaluation Group

Gino server and database                  Operation Communication & Interfaces

can handle
multiple arbitrary                 Flight Software                    Science Analysis
                                   Support Group                      Software Group
sequences of tasks:                                                                                                                LAT User
L1 pipeline; reprocessing;
MC; ….                              LAT Operations Steering Committee
                                               (Suggested)                                      Science Analysis                   Burst Alert
                                                                                                                                Messages via GCN

                                                                                                      LAT TEAM

                                                              Keep everything on disk and archive to tape
   Document: LAT-PR-04500                                                                     Section 3.4                         6
     GLAST LAT Project                                      ISOC CDR, 4 August 2004

                                Pipeline Spec
•   Function
     – The Pipeline has five major functions
          • automatically process Level 0 data through reconstruction (Level 1)
          • provide near real-time feedback to IOC
          • facilitate the verification and generation of new calibration constants
          • produce bulk Monte Carlo simulations
          • backup all data that passes through
•   Must be able to perform these functions in parallel
•   Fully configurable, parallel task chains allow great flexibility for use online as
    well as offline
     – Will test the online capabilities during Flight Integration

•   The pipeline database and server, and diagnostics database have been
    specified (will need revision after prototype experience!)
     – database: LAT-TD-00553
     – server: LAT-TD-00773
     – diagnostics: LAT-TD-00876

Document: LAT-PR-04500                                      Section 3.4          7
           GLAST LAT Project                     ISOC CDR, 4 August 2004

                               Pipeline in Pictures
                                                State machine +
                                                complete processing

Configurable                                               Expandable and
linked list of                                             configurable set of
applications                                               processing nodes
to run

      Document: LAT-PR-04500                     Section 3.4          8
     GLAST LAT Project                            ISOC CDR, 4 August 2004

                         Overall Test Plan

•   Combination of Engineering Model tests, Data Challenges and LAT
    Integration Support

•   EM tests
     – EM1 demonstrated ability to simulate/reconstruct real data from
       single (non-standard) tower
         • All within standard code framework/tools
         • Data analyzed with SAS tools
•   Data Challenges
     – End to end tests of sky simulation through astro analysis
     – Exercise pipeline

•   LAT Flight Integration
     – Combine tools from EM & DC applications
     – Sim/recon/analysis & pipeline processing and record keeping

Document: LAT-PR-04500                            Section 3.4          9
     GLAST LAT Project                                     ISOC CDR, 4 August 2004

                               Processing Flow

          Data Pipeline

                                                   Recon: interpret LAT readout
                  Level 0            Sim           and estimate directions and
                                                   energies; flag background

                                                   Sim: full modeling of e/g/p
                                                   interactions and readout in
                          Raw Data                             the LAT


                           Level 1         Science Tools

                                              Level 2

Document: LAT-PR-04500                                     Section 3.4            10
     GLAST LAT Project                                  ISOC CDR, 4 August 2004

                         Level 1 Sim/Recon Chain

                                             3 GeV g
                         Transport                 Real Data


                                                                     Particle ID

                                                                                  Level 1
Document: LAT-PR-04500                                 Section 3.4            11
     GLAST LAT Project                                ISOC CDR, 4 August 2004

                          Sim/Recon Toolkit
            Package          Description          Provider                Status
        ACD, CAL, TKR          Data                 LAT                 90% done
           Recon           reconstruction                                 In use
        ACD, CAL, TKR      Instrument sim           LAT                 95% done
             Sim                                                          In use
            GEANT4        Particle transport    G4 worldwide              In use
                                 sim            collaboration
               xml           Parameters        World standard             In use
               Root        C++ object I/O       HEP standard              In use
              Gaudi        Code skeleton       CERN standard              In use
            doxygen        Code doc tool       World standard             In use
         Visual C++/gnu   Development envs     World standards            In use
               CMT         Code mgmt tool       HEP standard              In use
             cvsweb        cvs web viewer      World standard             In use
               cvs        File version mgmt    World standard             In use

Document: LAT-PR-04500                                Section 3.4          12
     GLAST LAT Project                       ISOC CDR, 4 August 2004

               Software Development Approach
• Enable distributed development via cvs repository
• Extensive use of electronic communications
   – Web conferencing (VRVS), Instant Messaging (icq), Wikis
• CMT tool permits equal development on Windows and Linux
   – Coupled with cvs gives machinery enables config control
   – Superior development environment on Windows; compute
     cycles on linux
• documentation and coding reviews enforce coding rules
• JIRA issue tracking system
• “Continuous integration”
   – Eliminate surprises for incoming code releases
   – Build code every night; alert owners to failures in build or
     running of unit tests. Results tracked in database.
   – Developing comprehensive system tests in multiple source
     configurations. Track results in database; web viewable.

Document: LAT-PR-04500                      Section 3.4           13
     GLAST LAT Project                     ISOC CDR, 4 August 2004
                         Using Jira Issue Tracker

Document: LAT-PR-04500                     Section 3.4          14
          GLAST LAT Project                       ISOC CDR, 4 August 2004

                              Nightly Builds

  Performing builds
  for Science Tools
  now also

  Past release
                                    Display created from database query
                                                      Build status
Release in progress
                                                                Unit test status

     Future release

     Document: LAT-PR-04500                       Section 3.4          15
        GLAST LAT Project                  ISOC CDR, 4 August 2004

                            System Tests

Comparison of
current to previous

   Document: LAT-PR-04500                  Section 3.4          16
           GLAST LAT Project                                              ISOC CDR, 4 August 2004

                                    Main Science Tools

      Package                                 Description                                 First Use

     Likelihood         Workhorse model fitting for detection &
                        characterization of cosmic gamma-ray sources
  Level 1 database      Extracts desired event data
Exposure calculation    Uses IRFs, pointing, livetime etc. for deriving
                        calibrated source fluxes
Source identification   Identifies gamma-ray sources with cataloged
                        counterparts at other wavelengths
   GRB analysis         Temporal and spectral analyses of burst profiles                     DC1
  Pulsar analysis       Phase folding & period searching of gamma-ray
                        pulsars and candidates                                               DC2

    Observation         High level simulation of observations of the gamma-ray
     simulator          sky with the LAT

      Document: LAT-PR-04500                                              Section 3.4          17
     GLAST LAT Project                                  ISOC CDR, 4 August 2004

                         Science Tools Toolkit
          Package            Description         Provider                  Status

         PIL, PIL++        IRAF parameter       HEASARC                    In use
       cfitsio, CCFits       FITS file          HEASARC                    In use
       XSPEC, Sherpa      For GRB spectral    HEA standards          Under consideration
            Root               gui etc         HEP standard          Under consideration

           python             Scripting       World standard         Under consideration

          doxygen          Code doc tool      World standard               In use

       Visual C++/gnu     Development envs    World standards              In use

            CMT            Code mgmt tool      HEP standard                In use

           cvsweb          Cvs web viewer     World standard               In use

             cvs          File version mgmt   World standard               In use

Document: LAT-PR-04500                                 Section 3.4            18
             GLAST LAT Project                   ISOC CDR, 4 August 2004

                                 Data Products







        Document: LAT-PR-04500                   Section 3.4          19
        GLAST LAT Project                        ISOC CDR, 4 August 2004

                  Purposes of the Data Challenges
 • “End-to-end” testing of analysis software.                       S.Ritz

 • Familiarize team with data content, formats, tools and realistic
   details of analysis issues (both instrumental and astrophysical).

 • If needed, develop additional methods for analyzing LAT data,
   encouraging alternatives that fit within the existing framework.

 • Provide feedback to the SAS group on what works and what is
   missing from the data formats and tools.

 • Uncover systematic effects in reconstruction and analysis.

Support readiness by launch time to do all first-year science.
   Document: LAT-PR-04500                       Section 3.4           20
         GLAST LAT Project                                      ISOC CDR, 4 August 2004

                               SAS Checklist

Detailed Simulation          Instrument Calibration                 Processing Pipeline
Event Reconstruction              ACD    CAL       TKR
                                                                      MC      I&T
Event Classification

                               User Support

                               Code distribution                    High Level Instr Diags

                               Data Distribution                    Quicklook
 High Level Analysis

 GRBs,                                                              Institutional Mirrors
 Pulsars                                                            SSC
 Diffuse                                                            LAT Mirrors

                                        DC1        I&T EM
                                        DC2        I&T Flight
   Document: LAT-PR-04500                                       Section 3.4                 21
        GLAST LAT Project                                   ISOC CDR, 4 August 2004

                Data Challenge Planning Approach
• Walk before running: design a progression of studies.
• DC1. Modest goals. Contains most essential features of a data
  challenge. Original plan:
        •   1 simulated day all-sky survey simulation, including backgrounds
        •   find flaring AGN, a GRB
        •   recognize simple hardware problem(s)
        •   a few physics surprises
        •   exercise:
              – exposure, orbit/attitude handling, data processing pipeline components,
                analysis tools

• DC2, start end of CY04. More ambitious goals. Encourage further
  development, based on lessons from DC1. One simulated month.
• DC3. Support for flight science production.

   Document: LAT-PR-04500                                   Section 3.4          22
         GLAST LAT Project                                                    ISOC CDR, 4 August 2004

                                Data Challenge 1 Closeout
                                     12-13 Feb 2004

   Document: LAT-PR-04500                                                    Section 3.4           23
     GLAST LAT Project                             ISOC CDR, 4 August 2004

                         DC1 Components

•   Focal point for many threads
     – Orbit, rocking, celestial coordinates, pointing history
     – Plausible model of the sky
     – Background rejection and event selection
     – Instrument Response Functions
     – Data formats for input to high level tools
     – First look at major science tools – Likelihood, Observation Simulator
     – Generation of datasets
     – Populate and exercise data servers at SSC & LAT
     – Code distribution on windows and

•   Involve new users

•   Teamwork!

Document: LAT-PR-04500                             Section 3.4          24
         GLAST LAT Project                                                                                      ISOC CDR, 4 August 2004

         S.Digel and P.Nolan                         Science Tools in DC1 DC3

The big picture:                                                            Pulsar

Details are changing, but
                                                                         ephem. (D4)                 Standard Analysis Environment
still basically right                                  Pulsar period
                                                        search (A4)
                                                                         extract (U11)

          Level 0.5                  Event                                                                                  LAT Point source
                                                       Arrival time      Pulsar phase          Pulsar
                                 display (UI1)                                                                                catalog (D5)
                                                     correction (U10)    assign (U12)       profiles (A3)1

           Level 1 (D1)           Data extract                                                          Source model           Catalog
                                     (U1)                                                               def. tool (U7)                          Src. ID (A2)
                                                                                                                             Access (U9)
            Pointing/livetime                Exposure
            history (D2)         extractor (U3)                 calc. (U4)                                                      Astron.
                                                                                                       Likelihood (A1)       catalogs (D6)

            Alternative source     Alternative for                                                       Interstellar em.
            for testing high-      making additional                                                     model (U5)
            level analysis         cuts on already-
                                                                IRFs (D3)                                 Map gen
                                   retrieved event
                                                                                                           (U6)            User Interface aspects of the standard
                                                                                                                           analysis environment, such as
           Observation              Data sub-                                                          GRB unbinned        Image/plot display (UI2), Command
          simulator (O2)         selection (U2)                               IRF visual-           spectral analysis (A9) line interface & scripting (UI4), and
                                                                             ization (U8)                                  GUI & Web access (UI5) are not
                                                    GRB spectral-temporal shown explicitly.
          simulator (O1)         extractor (U3)                                                        modeling (A10)
                                                             GRB LAT DRM
                                                               gen. (U14)
                                                                                   GRB spectral           GRB visual-
  1                                                                                analysis (A8)2        ization (U13)
    This tool also performs periodicity tests and
  the results can be used to refine ephemerides                           GRB rebinning
  2 These tools can also take as input binned data                           (A6)2                      GRB temporal
                                                           GRB event
  from other instruments, e.g., GBM; the                  binning (A5)                                  analysis (A7)2
  corresponding DRMs must also be available.
 Document: LAT-PR-04500                                                                                        Section 3.4                        25
                                                                                                                                 14 Sept 2002
     GLAST LAT Project                                                                          ISOC CDR, 4 August 2004

     S.Digel and P.Nolan                      Science Tools in DC1

•   All components are still prototypes

      Level 1 (D1)           Data extract                                              Source model
                                (U1)                                                   def. tool (U7)

      Pointing/livetime             Exposure
      history (D2)          extractor (U3)              calc. (U4)
                                                                                      Likelihood (A1)

                                                                                        Interstellar em.
                                                                                                           The DC1 functionality is
       Alternative source     Alternative for
       for testing high-      making additional                                         model (U5)         Data extraction
       level analysis         cuts on already-
                              retrieved event           IRFs (D3)                        Map gen           Limited visualization
                              data                                                                         Model definition
      Observation              Data sub-                                                                   Model fitting
     simulator (O2)         selection (U2)
                                                                                                           Observation simulation
     simulator (O1)         extractor (U3)
                                                     GRB LAT DRM
                                                       gen. (U14)
                                                                     GRB spectral
                                                                     analysis (A8)2

                                                   GRB event
                                                  binning (A5)

Document: LAT-PR-04500                                                                         Section 3.4             26
     GLAST LAT Project              ISOC CDR, 4 August 2004

                         The data

                                         on to
Document: LAT-PR-04500              Section 3.4          27
             GLAST LAT Project                                                 ISOC CDR, 4 August 2004

• User’s Guide
• Data Description
• Likelihood Tutorial

DC-1 Discussion List

Analysis Code
download sites

                                                                                                Wiki page for
                                                                                                sharing results!

      Document: LAT-PR-04500                                                 Section 3.4            28
     GLAST LAT Project                             ISOC CDR, 4 August 2004

                         Lessons Learned

•   Analysis Issues                  •   Infrastructure Issues
     – Astrophysical data analysis        – SciTools did not run on
     – Software usage and                    windows at the last minute
       reliability                        – We discovered problems
     – Documentation                         with sources and ACD
     – Data access and data server           ribbons late
       usage                              – Manual handling of the
     – UI stuff                              processing
     – Software installation and          – No checking of file integrity
       release                            – Large failure rate in batch
     – Software infrastructure &             jobs (~10%)
       framework                          – Tools are not checking
     – Communication and Time                inputs much
       frame                              – Code distribution scripts
                                             were written manually

                                 Issues have been addressed

Document: LAT-PR-04500                             Section 3.4          29
        GLAST LAT Project                              ISOC CDR, 4 August 2004

                Strawperson Updated Plan for DC2
• DC2, based on lessons from DC1                                       S.Ritz
   –   1 simulated month of all-sky survey gammas (backgrounds: see next
   –   key sky addition: source variability
         • AGN variability, including bright flares, quiescent periods
         • expand burst variety (and include GBM? see later slides)
         • pulsars, including Gemingas, w/ orbit position effects.
   –   more realistic all-sky attitude profile
   –   background rate varies with orbit position
   –   more physics surprises, and add nominal hardware problems (and
       misalignments?), add deadtime effects and corrections

   – Analysis Goals:
        • produce toy 1-month catalog and transient releases
        • detailed point source sensitivity and localization studies
        • first systematic pulsar searches (timing!); detailed diffuse analyses
        • recognize simple hardware problems (connect with ISOC/SOG)
   – benchmark:
        • processing
   Document: LAT-PR-04500 times, data volume, data transfers.
                                                          Section 3.4    30
       GLAST LAT Project                                           ISOC CDR, 4 August 2004

                 SAS Support of Flight Integration
•     Expected Deliverables
       – Automated processing pipeline with bookkeeping
            • Gino: in test now
            • Revised draft pipeline spec at

       –   Stable Simulation/Reconstruction package
             • Frozen as of 8/4 (more versions will be necessary before actual use)
             • Special system tests for surface muon data

       –   Calibration algorithms for TKR, CAL – due by 1 Sept
            • TKR hot/dead strips
                  – Later: ToT gains, inter-tower alignment
             • CAL peds, gains, light tapers

       –   Calibration infrastructure for input to recon - rdbGui
            • Next step is allowing for binary format of parameters (800k ToT gains!)

       –   Event display – FRED
             • Ready

       –   Will inaugurate CCB process for code used in Pipeline

       –   Plan to extend System Tests scheme to deliver the High Level Diagnostics
Document: LAT-PR-04500                                             Section 3.4          31
     GLAST LAT Project                   ISOC CDR, 4 August 2004

                         Pipeline View

Document: LAT-PR-04500                   Section 3.4          32
     GLAST LAT Project                 ISOC CDR, 4 August 2004

                         Example of FRED

Document: LAT-PR-04500                 Section 3.4          33
     GLAST LAT Project                       ISOC CDR, 4 August 2004


• We believe that EMs, DCs and Flight Integration will leave us
  ready for flight

• EM1 worked with our tools

• DC1 worked well, showing very good capabilities from sky
  modeling through astronomical analysis
   – Plenty of work still to so, but reasonably understood
   – Will be demonstrated in DC2, 3 and LAT Integration, 16-
     tower cosmic ray tests and the beam test prior to launch

• LAT Flight Integration prep in full swing now

• DC2 within a year (being negotiated)

Document: LAT-PR-04500                       Section 3.4          34

To top