Data Acquistion Systems - Detector R_D - Fermilab

Document Sample
Data Acquistion Systems - Detector R_D - Fermilab Powered By Docstoc
					Data Acquisition Systems

          Ryan Rivera
 Fermilab Detector R&D Retreat
          May 5, 2011
                       Wikipedia

• Data acquisition
      – is the process of sampling signals that measure
        real world physical conditions and converting the
        resulting samples into digital numeric values that
        can be manipulated by a computer.




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   2
                                This Talk

                       1. Current DAQ program
                       2. Future projections
                         – Direction of Current Program
                         – Direction of New R&D




Detector R&D Retreat         Data Acquisition Systems - Ryan Rivera   3
                       Current Program

                        •      Optical Links
                        •      CAPTAN
                        •      xTCA
                        •      Mu2e




Detector R&D Retreat        Data Acquisition Systems - Ryan Rivera   4
                       Current Program

                        • Optical Links
                        • CAPTAN
                        • xTCA
                        • Mu2e




Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   5
                       Optical Links

     • Versatile Link Common Project
     • Free Space Optical Transmission




Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera   6
                       Optical Links

     • Versatile Link Common Project
     • Free Space Optical Transmission




Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera   7
        Versatile Link Common Project
            •   A CERN-organized common project for ATLAS and CMS.
            •   Charge: Develop general purpose optical link @ 5 Gbps data rates.
            •   Fermilab has back-end component responsibilities.
            •   Leading parallel optics investigations.




Detector R&D Retreat            Data Acquisition Systems - Ryan Rivera              8
                       Optical Transceiver Test
                           Measurements




                       Industry Standard Measurements and Apparatus
  Eye Diagram Measurements:                           Jitter Analysis:
       • Optical Modulation Amplitude                       • Deterministic Jitter Decomposition
       • Extinction Ratio                                   • Random Jitter (Gaussian, Unbounded)
       • Rise/Fall Times                                    • Eye Opening @ 10-12 BER

Detector R&D Retreat             Data Acquisition Systems - Ryan Rivera                     9
            CMS Pixel Optohybrid (POH)
• Status: Current laser no longer available
      – Upgrades will require new devices
• Requirements:
      – Rad hard
      – 640 Mbps
• Next: FNAL responsible for testing and
  characterizing devices


Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   10
  Parallel Optics – Technology Evolution




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   11
     Parallel Optics – Device Evaluation
                                                                           SNAP12 Transmitter
                                                                              (12 channels,
                                                                            2.7 Gbps/channel)




Parallel Optical Engine Transceiver                                      SFP+ Single Channel
 (4 channels, 6.25 Gbps/channel)                                             Transceiver
                                                                              (10 Gbps)

   Detector R&D Retreat         Data Acquisition Systems - Ryan Rivera                     12
                       Optical Links

     • Versatile Link Common Project
     • Free Space Optical Transmission




Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera   13
                       Free Space Optical Transmission
               • Motivation
                                                                                                                 10Gb/s Optical
                                                                                                                 Receivers

                        – Reduce material budget
                        – Work within rigid space constraints

                       Measured Transmission Spectrum of Silicon
                                   (IC grade doped)
               0.600                                                                         10 Gbps Optical
                                                                                             Transmitters at                      Silicon
                                                                                                   different                      Detectors
               0.500                                                                            wavelengths
                                                                                     ~100-150 cm
               0.400
Transmission




               0.300

               0.200                                                                     ~50-100 cm

               0.100

               0.000                                                                         ~10- 50cm
                    1400        1450        1500         1550         1600

                                       Wavelength (nm)                                                         Beam Line Center




               Detector R&D Retreat                      Data Acquisition Systems - Ryan Rivera                                        14
                                          Free Space
                                       Proof of Concept                                  l1 = 1470 nm
                                                                                         l2 = 1490 nm
CWDM: Coarse Wavelength Division Multiplexing                                            l3 = 1510 nm
                                                                                         l4 = 1530 nm

       CWDM DeMux                                    21 cm                          CWDM Mux
                                                     8cm


                                    Lens                              Lens
                                           ~2 mm Si        ~2 mm Si

           l1                                FPGA-Based BERT                                  l1
                         Rx                                                         Tx
           l2                                                                                 l2
                         Rx                                                         Tx
           l3                                                                                 l3
                         Rx                                                         Tx
           l4                                                                                 l4
                         Rx                                                         Tx
                              Rx LVDS Electrical               Tx LVDS Electrical
                              Bit Streams                      Bit Streams


     Next Step: Evolve the Optical Design for Detector Applications
  Detector R&D Retreat                Data Acquisition Systems - Ryan Rivera                        15
            Free Space Proof of Concept
                     Lab Test




       Operated error free for over 48 hours at 1 Gbps on all 4 four channels

Detector R&D Retreat         Data Acquisition Systems - Ryan Rivera             16
                       Optical Links Future
• Parallel optics
      – Initiated US R&D request to recent DOE detector
        solicitation. Need KA15 money until DOE approval.
      – Working with vendors on emerging devices, and
        SMU/OSU on custom devices.
• Versatile Link and CMS Phase 1 Optohybrid
      – Activities not funded by KA15, funded through CMS.
• Free space optics
      – Exploring low mass/power options (modulators).

Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera   17
                       Current Program

                        • Optical Links
                        • CAPTAN
                        • xTCA
                        • Mu2e




Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   18
                                 CAPTAN


                       What is the CAPTAN system?




Detector R&D Retreat         Data Acquisition Systems - Ryan Rivera   19
       Compact And Programmable daTa
              Acquisition Node




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   20
                       Attributes


• It is simple.
      – It is 6” x 6” and for many systems, the only
        external connections are a 3.3V power supply and
        a standard Ethernet cable.




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   21
                       Attributes


• It is flexible.
      – The user can stack the foundation boards in
        different combinations to give unique
        functionality.




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   22
                       Attributes


• It is scalable.
      – In addition to the vertical stacking, the stacks can
        be repeated arbitrarily and connected with one or
        many PCs in an Ethernet network.




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   23
                       User Template

   • The CAPTAN
   architecture consists of
   a few core boards but is
   intended to be
   augmented by custom
   boards designed and
   built by users.
                                                                 COOLING CHANNEL
                                                VERTICAL BUS


                                                LATERAL BUS      MOUNTING HOLE


                                                OPTICAL BUS      ELECTRONICS




Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera                     24
                         Core Boards
                       “Green Board”
                       NPCB – Node Processing
                       and Control Board


                                                                      “Blue Board”
                                                                      DCB – Data
                                                                      Conversion Board




                                                “Red Board”
                                                PDB – Power
                                                Distribution Board.



Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera                 25
                CAPTAN User Community
                           •     Fermilab
                           •     Brown
                           •     Purdue
                           •     Colorado
                           •     Milano
                           •     Lecce



Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   26
                       Applications
• The CAPTAN system was designed to handle
  common data acquisition, control, and processing
  challenges within high energy physics.
• Examples of such applications are tracker readout
  systems, R&D test stands, and parallel data
  processing.
• As the CAPTAN system is a modular system it can
  be used for a wide range of applications, from
  very small to very large.
Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   27
                       FTBF Pixel Telescope




Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera   28
                       PPD VIP Test Stand




Detector R&D Retreat      Data Acquisition Systems - Ryan Rivera   29
               QIE Irradiation BER Testing




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   30
                   IHEP Telescope - Beijing




Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   31
 T980 Crystal Collimation Telescope




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   32
                       CAPTAN Future
• Port CAPTAN Software to Linux

• Explore parallel
  processing power

• Integrate with xTCA



Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   33
                       Current Program

                        • Optical Links
                        • CAPTAN
                        • xTCA
                        • Mu2e




Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   34
                                 xTCA
• ATCA (Advanced Telecommunications
  Computing Architecture)
      – Spec put forth by PICMG (PCI Industrial Computer
        Manufacturers Group: a consortium of over 250
        companies)
• AMC plugs into ATCA card
      – Or MicroTCA crate
• ATCA + MicroTCA = xTCA
• Successor to VME?
                                                                12U 14-slot ATCA shelf

Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera                            35
   Advanced Mezzanine Card (AMC)




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   36
                       xTCA and CAPTAN
                          Integration




Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   37
                       VME vs. xTCA
      + Number of new developments decreasing, sales still
        constant
      + Many I/O modules available
      − Developed in 1981
      − Bus technology has speed limitations
      − Wide busses create a lot of noise in analog channels
      − No standard management at crate level
      − No standard management at module level
      − One damaged bus line stops entire crate
Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera   38
                        VME vs. xTCA
      + Scalable modern architecture
              From 1 slot MicroTCA to full mesh ATCA
      + 1 Gbps serial communication links
      + Standard PCIe and Ethernet communication+
        Well defined management
      + Hot-swappable
      + Safe against hardware/software failures
      + 99.999% availability is possible

Detector R&D Retreat      Data Acquisition Systems - Ryan Rivera   39
                        VME vs. xTCA
     − Many crates and modules from different
          manufacturers
             Is there a 'standard' ?
     − Learning curve to software/hardware
          specifications
     − Need functional MMC/IPMI code for power
          delivery
     − Imposed system management

Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera   40
                         xTCA Future
• Large experiments are considering xTCA over VME.
      –    CMS
      –    ATLAS
      –    LHCb
      –    Experiments at DESY
• Dream is to develop a readout system for strips/pixels
      – Evaluate for future FNAL Experiments (CMS upgrade,
        Mu2e)
• SBIR submitted with industry partners.
• Vince Pavlicek is on committee to develop the
  PhysMTCA or MTCA.4 (MicroTCA for Physics) spec.

Detector R&D Retreat      Data Acquisition Systems - Ryan Rivera   41
                       Current Program

                        • Optical Links
                        • CAPTAN
                        • xTCA
                        • Mu2e


Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera   42
                                      Mu2e
                                                Tracker
                                                     21,600 Straws
                                                     216 Readout Controllers
                                                     ~65 GBps*




                                                              Calorimeter
  * zero-suppressed data                                           2112 Crystals
                                                                   44 Readout Controllers
  - Additional data (< 1 GBps) from                                ~15 GBps*
    Cosmic Ray Veto system and
    extinction/target monitors


Detector R&D Retreat          Data Acquisition Systems - Ryan Rivera                        43
                       Architecture Alternatives
• Streaming (this is the selected architecture)
      + less front-end hardware (no concentrators or L1 Trigger)
      + simpler design
      + flexible trigger
      − higher cost
• Triggered
      + less back-end hardware (networking and processing)
      + lower cost
      + allows higher digitization rate (e.g., waveform digitizers)
      − fixed L1 trigger

Detector R&D Retreat         Data Acquisition Systems - Ryan Rivera   44
                               Implementation
                                 Alternatives
                                                   Servers
                                                 NIC                            Standalone
                  EVB Switch                                                   (rack-mount)
                                                                                    DTC
                       DTC

                   Servers                       EVB Switch
                 DTC                                                             PCIe DTC




                   Servers                        EVB Switch
                 DTC     NIC                                                     PCIe DTC
                                                                         (event building in server)



Detector R&D Retreat            Data Acquisition Systems - Ryan Rivera                        45
                                          Comparing
                                          Technology
• To meet the demands of a 100 GBps streaming
  DAQ, Mu2e will leverage…
      –    10 Gbps Networking
      –    High Density FPGAs
      –    High Performance Processors
      –    Fiber-optic Communications (synergy with Optical
           Links)
                                           Historical Perspective
                                          CDF DAQ (1987)        Mu2e DAQ
                          Data Rate             20               100,000           MBps
                          Buffering             400             2,000,000           MB
                       Processing (FP)          10             30,000,000          MFLOPS
                       Processing (INT)         200            60,000,000           MIPS


Detector R&D Retreat                      Data Acquisition Systems - Ryan Rivera            46
                         Design Philosophy
• Commercial Slow Control hardware and software is used for
  development
    – needed early, can’t wait for custom HW/SW
    – hopefully, utility extends into production and running
    – LabVIEW or EPICS

• Embedded Processors
    – use commercial modules if possible to allow early software development
    – embedded Ethernet and HTTP with eye towards debugging ease




  Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera        47
                       Mu2e Future
• One year from starting prototype phase.
• ~6 years to installation complete.
• Mu2e funded by Mu2e, but components of
  DAQ architecture may apply to future
  experiments (Project X and others).




Detector R&D Retreat    Data Acquisition Systems - Ryan Rivera   48
                       DAQ Discussion Points


                       • Program Resources
                       • National Picture Comparison
                       • Best Future R&D Avenues




Detector R&D Retreat          Data Acquisition Systems - Ryan Rivera   49
         DAQ Program Resource Issues
• Limited manpower
      – Has hindered xTCA effort
• xTCA development accessibility
      – Community MMC design effort?
• FPGA power/expense/expertise




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   50
          Summary of National Picture vs.
               Current Program
• Breaking new ground with optics, free space
      – Submitted SBIR and DOE R&D proposal
• Optics: NTT (Nippon Telegraph and Telephone) achieved 69.1
  Tbps transmission using WDM of 432 wavelengths with a
  capacity of 171 Gbps over a single 240 km optical fiber on
  March 25, 2010. Highest optical transmission speed ever
  recorded at time.
• xTCA falling behind?
• GPUs falling behind?
• DAQ Software falling behind?
      – COMEDI, EPICS, LabVIEW, MATLAB, Visual C++, ladder logic?

Detector R&D Retreat       Data Acquisition Systems - Ryan Rivera   51
                       Best Future DAQ
                        R&D Avenues
• Ethernet/InfiniBand: InfiniBand prevalent as the interconnect
  between nodes of the highest performing super computers
• Off-the-shelf components
• xTCA
• GPUs: Fastest computer in the world, Tianhe-1A in China uses 7,168
  NVidia Fermi GPUs and 14,336 Intel Xeon CPUs. Would require
  50,000 CPUs for same performance with CPUs alone.
• Optics: Commonly 100 Gbps with distances over 100 m. Becoming
  interconnect of choice in high speed data centers and HPC clusters.
  There are 40 Gbps and 100 Gbps optical Ethernet standards.
• FPGAs
• EPICS: The tool is designed to help develop systems which often
  feature large numbers of networked computers providing control
  and feedback.


Detector R&D Retreat     Data Acquisition Systems - Ryan Rivera    52
                           Thanks!




Detector R&D Retreat   Data Acquisition Systems - Ryan Rivera   53

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:6
posted:2/22/2013
language:English
pages:53