The Computational Microscope

Document Sample
The Computational Microscope Powered By Docstoc
					                        The Computational
                               NAMD User Experience on Abe
                                          Klaus Schulten
                            Dept. Physics / Beckman Institute, U. Illinois

                                                  of an en   tire virus
1                                   s   imulation

     Classical Molecular Dynamics
                       F=ma at 300K
    Energy function:

    used to determine the force on each atom:

    yields a set of 3N coupled 2nd-order differential equations
    that can be propagated forward (or backward) in time.

    Initial coordinates obtained from crystal structure,
    velocities taken at random from Boltzmann distribution.

    Maintain appropriate temperature by adjusting velocities.
                Classical Dynamics
         discretization in time for computing

Use positions and accelerations at time t and the positions
from time t-!t to calculate new positions at time t+!t.


      Potential Energy Function of
•   Simple, fixed algebraic form for every type of interaction.
•   Variable parameters depend on types of atoms involved.


                                                                  from physics
                   The Holy Grail of MD!
            biomolecular timescale and timestep limits
                                   s         1015
  Rotation of buried sidechains
           Local denaturations
          Allosteric transitions   ms        1012

                                   µs        109

                 Hinge bending     ns        106
Rotation of surface sidechains
                                             (day)                                              SPEED
             Elastic vibrations                                                                  LIMIT
                                   ps        103
               Bond stretching                                                                  !t = 1 fs
  Molecular dynamics timestep      fs        100

                              : A Computational Microscope

     Four Highlighted Features                    2002
1.    Ground Breaking                   Gordon Bell Award Winner
2.    Active User Base
3.    Extremely Scalable
4.    “Harnessing Parallelism for
      Small and Fast Simulations”

                                        NAMD: Biomolecular Simulation on Thousands of Processors
                                        Authors: James C. Phillips, Gengbin Zheng, Sameer Kumar, Laxmikant V. Kale,
                                        University of Illinois at Urbana-Champaign.
                                        Researchers achieved unprecedented scaling of NAMD, a code that
                                        renders an atom-by-atom blueprint of large biomolecules and
                                        biomolecular systems.

                                : A Computational Microscope
                                                                                                                        NAMD Registrants
                                                                                                                     19,995 Registrants (3336 NIH)
     Four Highlighted Features                                                                                            4,111 Repeat Users
1.    Ground Breaking Simulations

                                                                             Registered NAMD Users
                                                                                                                  NAMD 2.6 released Aug 2006
2.    Active User Base                                                                                           4181 NAMD 2.6 users (742 NIH)
3.    Extremely Scalable                                                                             20,000

4.    “Harnessing Parallelism for
      Small and Fast Simulations”

                                                                                                               02       03            04               05       06          07


                                : A Computational Microscope
     Four Highlighted Features                                                                           NAMD
                                         Simulation Rate in Nanoseconds Per Day

1.    Ground Breaking Simulations                                                      10                scales
2.    Active User Base                                                                                   by 103
3.    Extremely Scalable
4.    “Harnessing Parallelism for                                                   0.1
                                                                                                                                                            LYSOZYME (40K atoms)
                                                                                                                                                            APOA1 (92K atoms)
                                                                                                                                                            ATPase (327K atoms)
      Small and Fast Simulations”                                                                                                                           STMV (1M atoms)
                                                                                                                                                            BAR d. (1.3M atoms)
                                                                                                                                                            IAPP (5.5K atoms)
                                                                                                     1          10           100                1000         10000          100000

                                                                                                         "We haven't found a hard limit
                                                                                                         on scaling up the number of
           NAMD scales by 103
  IAPP                                                                                                                -- Philip Blood and Greg Voth,
                                                                                                                                  Univ Utah
                                                                                                            Commenting on NAMD performance
                                  STMV                                                                      for the PSC XT3 Cray
                              : A Computational Microscope
                                            dihydrofolate reductase        DHFR Benchmark
                                                                                23,558 atoms
                                                                             2489 protein atoms
 Four Highlighted Features                                                   21,069 water atoms
                                                                                1 fs timestep
1. Ground Breaking                                                          62.23 Å3 System size
                                                                          64 x 64 x 64 PME Grid
   Simulations                                                             9 Å non-bonded cutoff
                                                                          NVE Ensemble, 300 K
                                                              MTX      Bonds to Hydrogen constrained
2. Active User Base                                                            92 atoms / core

3. Extremely Scalable
4. “Harnessing Parallelism
   for Small and Fast                                                           31 ns/day
                                                                            on 256 processors


      NAMD: Practical Supercomputing
  •   20,000 users can’t all be computer experts.
       – 18% are NIH-funded; many in other countries.
       – 4200 have downloaded more than one version.
  •   User experience is the same on all platforms.
       –   No change in input, output, or configuration files.
       –   Run any simulation on any number of processors.
       –   Automatically split patches and enable pencil PME.
       –   Precompiled binaries available when possible.
  •   Desktops and laptops – setup and testing
       – x86 and x86-64 Windows, PowerPC and x86 Macintosh
       – Allow both shared-memory and network-based parallelism.
  •   Linux clusters – affordable workhorses
       – x86, x86-64, and Itanium processors
       – Gigabit ethernet, Myrinet, InfiniBand, Quadrics, Altix, etc
          Simulation of the Entire Ribosome
      Collaborators: Joachim Frank (HHMI), Willy Wriggers (UT Houston)


                                                                                                       IAPP (islet amyloid peptide)


                                                   Simulation Rate in Nanoseconds Per Day
                                                                                              1                                                      scales
                                                                                                                                                     by 103
                                                                                                                                 IAPP (5.5K atoms)

- 2.8 million-atom simulation of                                                             0.1                                 LYSOZYME(40K atoms)
                                                                                                                                 APOA1 (92K atoms)
ribosome with NAMD2.6 (Core 3)                                                                                                   ATPASE (327K atoms)
                                                                                                                                 STMV (1M atoms)
- 2,048 processors on a Cray XT3                                                                                                 BAR D (1.3M atoms)
                                                                                                                                 RIBOSOME (2.8M atoms)
- 3ns / day                                                                                 0.01

- Pencil decomposition for PME                                                                     1         10        100
                                                                                                                                      1000             10000

provided a 20% speedup

      NAMD: Designed for Clusters
•     Resource has pioneered clusters for MD:                                                                      June 2001
       – 1993: HP workstation cluster, 12 nodes
            • Original NAMD development platform.
       – 1998: Pentium II Linux cluster, 16 nodes
       – 2001: Athlon Linux clusters, 3x32 nodes
            • Used for first cluster workshop in June.
            • On-demand queues for short jobs.
       – 2003: Dual-Athlon clusters, 3x24 nodes
       – 2004: Dual-Xeon clusters, 3x24 nodes
       – 2005-2006: Five hands-on workshops                                                                       April 2006
            • 24 students per workshop.
            • Students build clusters in teams of four.
•     Most popular platform in 2005 NAMD user survey was
      “Local Linux cluster.”
             NAMD Shines on InfiniBand
                       Fall 2006: TACC Lonestar comes online.
                         Commodity cluster with 5200 cores!
             100                                                     32 ns/day
                                                                    2.7 ms/step
                                                    m s)
                                            k   ato
              10                      R (24                                    15 ns/day
ns per day

                             C /DH            ms
                                                )                             5.6 ms/step
                           JA              ato
                                        2k                  s)
                                    1 (9              a tom
               1                  oA
                                Ap                (1M
                                           S TM
                       4        8    16   32     64        128 256 512 1024

    Porting NAMD to Abe Cluster
• Build on experience from TACC Lonestar
             – Serious performance bugs in MVAPICH 0.9.8
                    • NAMD/Charm++ stresses MPI with “unexpected” messages
             – Fixed with 0.9.9 beta 2 at TACC
• On Abe use MPICH-VMI instead
             – Fast and stable for most users
             – Million-atom simulations have network errors
• Work in progress
             – Native InfiniBand ibverbs port of Charm++
             – Actually try MVAPICH 0.9.8 on Abe
             – Detailed performance debugging at limits of scaling
                   NAMD on NCSA Abe
              NCSA Abe is based on Dell servers and InfiniBand.
             Commodity cluster with 1200 blades and 9600 cores!
             100                                                   26 ns/day
                                                     ms)          3.3 ms/step
                                            k   ato
             10              H     FR     s)
ns per day

                         C/D         atom
                       JA        92k                   s)
                              1 (                a tom
                          ApoA               (1M                                 4.7 ns/day
              1                          MV                                      18 ms/step
                                      ST                                        7 cores/node

                   8      16       32     64     128 256 512 1024 1792

      NCSA Abe vs TACC Lonestar
              Both clusters based on Dell servers and InfiniBand.
             Difference is quad-core (Abe) vs dual-core (Lonestar).
             100                                                   32 ns/day
                                                     ms)          2.7 ms/step
                                            k   ato
                                 R      (24                                        15 ns/day
             10                HF                   s)                            5.6 ms/step
ns per day

                       JAC           2k
                                                a                   ms)
                                 1 (9                           ato
                               oA                        ( 1M
                             Ap                       MV
                                                                                 4.7 ns/day
                                                ST                               18 ms/step
                                                                                7 cores/node

                   4     8        16    32      64    128 256 512 1024 1792
   NAMD projects run well on Abe
      polio virus                                        nanopore
                  System                  Size (atoms)   CPUs   Benchmark(ns/day)   SUs used
                  Apolipoprotein A-1      98,000         96     3.9                 12,000
                  Discoidal HDL           144,000        128    3.6                 12,800
                  STMV                    1,080,000      384    0.9                 64,300
                  Lipase                  34,000         384    64.0                45,000
                  Silica Nanopore         46,000         32     2.9                 24,000
                  FD coat protein         57,000         192    13.0                10,000
                  Translocon SecY         106,000        128    5.0                 30,000
                  Chromatophore LH2       910,000        256    1.2                 49,100
                  Poliovirus              1,000,000      1024   2.5                 49,200
                  TOTAL                                                             296,400


                                       Molecular Basis of Hearing

                                                                            Inner Ear Mechanism

                                                                                    M. Sotomayor, D.
                                                                                    Corey, and K.
                                                                                    Schulten Structure,
                                                                                    13:669-682, 2005
Tip Links (Kachar       Hair bundle (Assad
et al., 2000; Corey     and Corey, from Sensory
Lab)                    Transduction, G. L. Fain).
                                            Molecular Basis of Hearing

                                                                                  Inner Ear Mechanism

Tip Links (Kachar           Hair bundle (Assad
et al., 2000; Corey         and Corey, from Sensory
Lab)                        Transduction, G. L. Fain).

         Molecular dynamics simulations of complete C-cadherin
         extracellular domain in water with and without calcium:
                             345,000 atoms
                                                     • Crystal Structure of C-cadherin from Boggon et
                                                     al. 2002 (pdb code 1L3W).
                                                     • >100 ns of combined simulation time using
                                                     multiple supercomputers.
                                                     • Abe: 0.38 days/ns (2.6 ns/day) using 344
                          +Ca2+                                                  -Ca2+

    •1L3W with five modules (without sugars) + 12                  •1L3W with five modules (without sugars) no
    calcium ions + 2 disulfide bonds on EC5 +                      calcium ions + 2 disulfide bonds on EC5 +
    Crystallographic Water                                         Crystallographic Water
             –345K atoms total, 12 Ca+, 8 Na+, 364x105x94Å               – 345K atoms total, 32 Na+, 364x105x94Å
    •Equilibration (NpT, PBC, PME, CMAP,dt=1,1,1)                  •Equilibration (NpT, PBC, PME, CMAP,dt=1,1,1)
          –Protein restrained (100ps)                                    – Protein restrained (100ps)
          –Protein released (1ns NpT + 4ns in NVE)                       – Protein released (1ns NpT + 4ns in NVE)
 Calcium Ions Protect Cadherin from Unfolding

Steered Molecular Dynamics (k=1kcal/mol/Å2; v=0.1 or 0.01Å/ps )

                  Biomedical Studies of Lipoproteins

                                      •   Residue-based coarse-graining developed to simulate
                                          nanodisc assembly & disassembly
                                      •   CG Builder: coarse-graining and reverse coarse-graining
                                      •   Compare simulation results with experimental SAXS
 How do nanodiscs self-assemble?      •   Current biomedical applications include experimental
                                          studies of membrane bound
                                           –    Cytochrome P450
                                           –    G-Protein Coupled Receptors
                                           –    Blood Clotting Tissue Factors
                                           –    High-Density Lipoproteins

                                                                                Collaboration with
                                                                                Steve Sligar

    Nanodisc Assembly              SAXS at APS, Argonne, IL
                  High-density lipoproteins
                                                 •    Discoidal HDL: two apolipoprotein A-I
                                                      wrapped around a POPC bilayer
                                                       – 144,000 atom system equilibrated for
                                                          15 ns
                                                       – 3.6 ns per day using 128 NCSA Abe
                                                          CPU’s (16 nodes)

  •     Lipid-free apolipoprotein A-I
         – 98,000 atom system equilibrated for
           20 ns
         – 3.9 ns per day using 96 NCSA Abe
           CPU’s (12 nodes)

Studying the Physical Mechanism of Photosynthesis

                                                     Describe entire living
                                                        cells at atomic
light                                                     resolution

                               ADP         ATP
Studying the Physical Mechanism of Photosynthesis

                                          How does membrane
light                                         curvature

                          ADP       ATP

         Protein Packing Induces Membrane
   Seven Rb. sphaeroides peripheral light
   harvesting (LH2) complexes in mixed
   POPE/POPG membrane patch                  top view

        909830 atoms
        ~10 ns full equilibration

        ~ 1.2 ns/day on 32 Abe nodes
        NPT equilibration, Charmm
        force field
                                             side view
                 Mechanism of Poliovirus Entry
              Collaboration with J. Hogle and X. Zhuang (Harvard)

In vivo:
             virus,                     Swollen and
            “160S”                      partially open
             state       RNA            “135S” state                       “80S” state


                                                                               RNA release

In vitro:
    160S              T ~> 37 C

     +                  low T

   Studying the Initial Transition in Poliovirus
      Capsid (160S " 135S): Structures
                                                                      All-atom model
Poliovirus+Receptors: cryo-EM
                                                                   (160S with receptors)

                           Homology model of the receptor fitted
                                into the cryo-EM density
         46 nm
Obtaining the 135S structure by fitting 160S into the 135S cryo-EM map:

                                   Real dynamics and
                                interactions in between?

                                   “Abe” to the rescue!
            Initial                                                        After the fitting
  Studying the Initial Transition in Poliovirus
 Capsid (160S " 135S): Dynamics on “Abe”
                                                 System size: 1,000,000 atoms

                                                 Simulated time: 5 ns (and counting)

                                                 Processors: 1024 (128 nodes)

                                                 Performance: 2.5 ns/day

                                                 Simulation detail: NpT ensemble,
                                                 CHARMM forcefield, NAMD

                                                Results of this simulations, in
                                              conjunction with the160S"135S
                                                fitting, will be used to identify
                                              the capsid-receptor interactions
Segment of the capsid                           responsible for the transition
  with 5 receptors

              Satellite tobacco mosaic virus (STMV)

STMV - 60 protein icosahedron around an RNA
• 1,079,000 atom system
• Six 1-ns pulling simulations performed on
• 0.82 ns per day using 384 NCSA Abe CPUs
   (48 nodes)

                                     •   Steered molecular dynamics (SMD) applied to
                                         study stabilizing interactions
                                     •   SMD at different protonation states used to probe
                                         pH-driven particle disassembly
                                     •   Results so far highlight importance of protein-
                                         RNA interactions in the protein’s
                                         N-terminal “tail”
                                     •   Ongoing simulations on abe to further
                                         characterize these interactions
    DNA Recording with Synthetic Nanopores
     Electronic decoding of DNA sequence using synthetic
     A low cost, rapid DNA sequencing method would significantly
     impact life science and personal medicine
     Integrates both biomolecular and inorganic structures via methods
     extensible to more complex systems

                         Wong & Pettitt,
                         Biopolymers 73: 570-578 (2004)
                         an early treatment!

Current rectification in silica nanopores
                                                          •   Asymmetric current-voltage curves (so-
                                                              called current rectification) are observed in
                                                          •   The reason of the asymmetry was not clear.
                                                              No experimental technique can visualize the
                                                              ion dynamics inside the nanopore.
                                                          •   MD simulations act as microscopes, reveling
                                                              that current rectification is due to ion
                                                              adsorption on the nanopore surface

                   •   System size: 46000 atoms.
                   •   8 different MD runs of 10 ns                                           Ideal
                   •   2.7 ns/day using 32 NCSA                                                  Simulation
                       Abe CPU’s (4 nodes)                          Current rectification
       NCSA Abe user experiences
• Positive
   – NAMD scales well
       106K atoms: 128 CPU (100%), 256 CPU (81%), 512 CPU(73%)
   – Fast computing
   – Short queues (large jobs run quickly!)
• Negative
   – Multiple crashes upon submission experienced by large jobs
   – Jobs that have “died” continue to be listed as running even though no
     output is being generated
   – Hardware failure resulting in inaccessible scratch space

Shared By: