The Read-Out Crate in ATLAS

Document Sample
The Read-Out Crate in ATLAS Powered By Docstoc
					  The VMEbus processor hardware
   and software infrastructure in

                  Markus Joos, CERN-PH/ESS

11th Workshop on Electronics for LHC and future Experiments,
        12-16 September 2005,Heidelberg, Germany
   LECC 2005, Heidelberg                 Markus Joos, CERN-PH/ESS
  VMEbus in ATLAS front-end DAQ
                                                   ATLAS VMEbus crates:
                                                    VME64X
                    Event data,
                                                    6U or 9U
~100 * 9U        Timing and trigger
                                       ~55 * 6U     Air or water cooled
                                                    Local or remote Power supply
                                                    Connected to DCS via CAN

                                                  One type of VMEbus master
                             TTC and auxiliary
                             modules, some
                                                  for all crates:
                             RODs                   To bring down price by
                                                     economy of scale
RODs and some                             TTC
                                                    To ease maintenance and
modules                   Event data                 provision of spares
                                                    To avoid S/W
    Event data
                    ROS PCs            LvL2          incompatibilities

     LECC 2005, Heidelberg                                   Markus Joos, CERN-PH/ESS
    Basic choice: Embedded SBC or Link to PC
Main requirements
 Mechanical: 6U, 1 slot, VME64x mechanics
 VMEbus protocol: Support for single cycles, chained DMA and interrupts
 VMEbus performance: At least 50% of the theoretically possible bus transfer rates
 Software: Compatibility with CERN Linux releases

These requirements can be met by both an embedded SBC or an interface to a PC system

SBC                                              Link to PC
                                                  Computing performance can be increased
 Space conservative (important in ATLAS
                                                    by faster host PC
  underground area)
                                                  Possibility to control several VMEbus
 Typically better VMEbus performance               crates from one PC
  (especially single cycles)
 Cheaper (system price)

         SBC better suited for the requirements of ATLAS

      LECC 2005, Heidelberg                                     Markus Joos, CERN-PH/ESS
          Finding an appropriate SBC
Type of CPU
Intel                                     PowerPC
 Higher clock rates                       Big endian byte ordering (matches VMEbus)
 Better support for (CERN) Linux          Vector processing capability

       Let the market decide

Other requirements (just the most important ones)
   Support for diskless operation
   At least 256 MB main memory
   One PMC site
   One 10/100 Mbit/s Ethernet interface
   VME64 compliant VMEbus interface
   8 MB flash for Linux image
   Repair service for at least 10 years
   Guaranteed H/W evolution and backwards compatibility

      LECC 2005, Heidelberg                                  Markus Joos, CERN-PH/ESS
            Pitfalls – the Universe chip
Most of today’s SBCs are based on the Tundra Universe chip which was designed
in ~1995. Despite some improvements (latest version is Universe II D) it still has a
few shortcomings:

 Lack of built in endian conversion
     Intel based SBCs have to have extra logic to re-order the bytes
 Performance
     Single cycles: ~ 1 μs (arbitration for three busses)
     Block transfer: ~60% of theoretical maximum (i.e. 25 MB/s for D32, 50 MB/s for
 VMEbus bus error handling
     Errors from posted write cycles are not converted to an interrupt
 Lack of constant address block transfers for reading out FIFOs
     My be required to read out a FIFO based memory
     It is possible to have the Universe chip do it with (slow) single cycles (~13 MB/s)
     Danger of loosing last data word on BERR terminated transfers

      LECC 2005, Heidelberg                                      Markus Joos, CERN-PH/ESS
                             Other issues
 Concurrent master and slave access
    If a SBC is VMEbus master and slave at the same time deadlock situations on PCI are
      possible. Some host bridges resolve them by terminating the inbound VMEbus cycles
      with BERR
 Remote control and monitoring
    Most SBCs do not support IPMI yet
    Some vendors put temperature or voltage sensors on their SBCs but there is no
      standard way of reading this information
    Remote reset: Via 2-wire connector (in the front panel) or SYSRESET*
 Mechanics
    VME64x alignment pin and P0 are incompatible with certain crates. Some vendors
      provide alternative front panels or handles.
    EMC gaskets can be “dangerous” for solder side components on neighboring card.
      Use solder side covers

     LECC 2005, Heidelberg                                   Markus Joos, CERN-PH/ESS
     What ATLAS finally has chosen
 In 2002 a competitive Call for Tender was carried out
     Non-technical requirements:
         3 years warranty
         10 years support
         Fixed price for repair or replacement
         Controlled technical evolution
     All major suppliers contacted
     ~10 bids received
 Features of the default SBC that ATLAS has chosen
     800 MHz Pentium III
     512 MB RAM
     Tundra Universe VMEbus interface
     ServerWorks ServerSet III LE host bridge
 Since 2005 a (software compatible) more powerful SBC is available as an alternative
     1.8 GHz Pentium M
     Up to 1 GB RAM
     KVM
     Two GB Ethernet ports
     Intel 855 GME host bridge

     LECC 2005, Heidelberg                                  Markus Joos, CERN-PH/ESS
        Choice of the operating system
 Unix compatible environment (to leverage existing experience)
 No hard real time
     Interrupts are used in some applications but their latency is not crucial
 Full local developing and debugging environment
 Cheap (ideally no) license costs

        Linux does it all
 The ATLAS default SBC uses the SLC3 release
 Only minor modifications to the kernel configuration to support diskless operation
 “Look and feel” as on a Linux desktop PC (X11, AFS, etc.)

      LECC 2005, Heidelberg                                       Markus Joos, CERN-PH/ESS
                              VMEbus S/W
 Analysis of third party VMEbus I/O libraries:
    Design
          There is (despite VISION, ANSI/VITA 25-1997) no standard API for VMEbus
          The I/O libraries provided by the board manufacturers have incompatible APIs
          Some libraries (unnecessarily) expose details of the underlying hardware to the
    Implementation
          Tradeoff between speed and functionality may not suit user requirements
    Completeness
          Sometimes less frequently used features are not supported
    Support
          Not always guaranteed (especially for freeware)

       We decided to specify our own API and to implement a driver & library

      LECC 2005, Heidelberg                                    Markus Joos, CERN-PH/ESS
             The ATLAS VMEbus S/W
 Linux allows the development of an I/O library almost fully in user space
     Highest speed (no context switches)
     Multi tasking and interrupt handling difficult
 Our solution:
     Linux device driver
           Support for multi tasking
           SMP not an issue (SBC has one CPU, no Hyper-threading)
           Interrupt handling
           Block transfers
     User library
           C–language API
           Optional C++ wrapper
     Comprehensive S/W tools for configuration, testing, debugging and as coding
     Independent package for the allocation of contiguous memory (for block transfers)

      LECC 2005, Heidelberg                                    Markus Joos, CERN-PH/ESS
 Main features of the ATLAS VMEbus
             I/O package
 Single cycles
     Static programming of the PCI to VMEbus mapping
     Fast (programmed I/O from user code)
     Safe (full synchronous BERR* checking)
     Special functions for geographical addressing (CR/CSR space access)
 Block transfers
     Support for chained DMA
     Fixed address single cycle block transfers for FIFO readout
 Interrupts
     Support for ROAK and RORA
     Synchronous or asynchronous handling
     Grouping of interrupts
 Bus error handling
     Synchronous or on request
 Performance
     Fast single cycles: 0.5 to 1 μs
     Safe single cycles: 10 to 15 μs
     Block transfers (overhead): 10 to 15 μs

     LECC 2005, Heidelberg                                   Markus Joos, CERN-PH/ESS
   Supply contract for ATLAS established in 2003
   So far ~170 SBCs purchased
       Successfully used in 2004 ATLAS combined test-beam
       Many small test-benches and laboratory systems
       Installation in final ATLAS DAQ system started
   The CERN Electronics Pool is phasing out the previous
    generation of PowerPC / LynxOS based SBCs in favor of the
    ATLAS model
   The ALICE Experiment will use the same model in the VMEbus
    based part of the L1 trigger system

LECC 2005, Heidelberg                          Markus Joos, CERN-PH/ESS
 ATLAS had no special requirements (such as e.g. 2eSST support)
  on the SBC
 The time spent on the evaluation of SBCs was well invested. Some
  important lessons have been learnt and helped with the Call for
 Specifying our own VMEbus API and implementing the software
  from scratch paid out in terms of flexibility and performance
 Both the SBC and the software have been used successfully in a
  large number of systems

  LECC 2005, Heidelberg                         Markus Joos, CERN-PH/ESS
I would like to thank:
 Chris Parkman, Jorgen Petersen and Ralf Spiwoks for their
  contribution to the technical specification of the ATLAS SBC and
  VMEbus API
 Jorgen Petersen for his assistance with the implementation of the
 The members of the ATLAS TDAQ team for their contributions
  and feed back

  LECC 2005, Heidelberg                          Markus Joos, CERN-PH/ESS

Shared By: