The Computational Microscope

Document Sample
The Computational Microscope Powered By Docstoc
					                        The Computational
                           Microscope
                               NAMD User Experience on Abe
                                          Klaus Schulten
                            Dept. Physics / Beckman Institute, U. Illinois




                                                  of an en   tire virus
1                                   s   imulation




     Classical Molecular Dynamics
                       F=ma at 300K
    Energy function:

    used to determine the force on each atom:



    yields a set of 3N coupled 2nd-order differential equations
    that can be propagated forward (or backward) in time.

    Initial coordinates obtained from crystal structure,
    velocities taken at random from Boltzmann distribution.

    Maintain appropriate temperature by adjusting velocities.
                Classical Dynamics
         discretization in time for computing



Use positions and accelerations at time t and the positions
from time t-!t to calculate new positions at time t+!t.


                                                                         +




      Potential Energy Function of
              Biopolymer
•   Simple, fixed algebraic form for every type of interaction.
•   Variable parameters depend on types of atoms involved.



                                                                  heuristic




                                                                  from physics
                   The Holy Grail of MD!
            biomolecular timescale and timestep limits
                                        steps
                                   s         1015
  Rotation of buried sidechains
           Local denaturations
          Allosteric transitions   ms        1012

                                   µs        109
                                             (year)


                 Hinge bending     ns        106
Rotation of surface sidechains
                                             (day)                                              SPEED
             Elastic vibrations                                                                  LIMIT
                                   ps        103
               Bond stretching                                                                  !t = 1 fs
  Molecular dynamics timestep      fs        100




                              : A Computational Microscope


     Four Highlighted Features                    2002
1.    Ground Breaking                   Gordon Bell Award Winner
      Simulations
2.    Active User Base
3.    Extremely Scalable
4.    “Harnessing Parallelism for
      Small and Fast Simulations”


                                        NAMD: Biomolecular Simulation on Thousands of Processors
                                        Authors: James C. Phillips, Gengbin Zheng, Sameer Kumar, Laxmikant V. Kale,
                                        University of Illinois at Urbana-Champaign.
                                        Researchers achieved unprecedented scaling of NAMD, a code that
                                        renders an atom-by-atom blueprint of large biomolecules and
                                        biomolecular systems.


       19
                                : A Computational Microscope
                                                                                                                        NAMD Registrants
                                                                                                                     19,995 Registrants (3336 NIH)
     Four Highlighted Features                                                                                            4,111 Repeat Users
1.    Ground Breaking Simulations




                                                                             Registered NAMD Users
                                                                                                                  NAMD 2.6 released Aug 2006
2.    Active User Base                                                                                           4181 NAMD 2.6 users (742 NIH)
3.    Extremely Scalable                                                                             20,000

4.    “Harnessing Parallelism for
      Small and Fast Simulations”
                                                                                                     10,000



                                                                                                           0
                                                                                                               02       03            04               05       06          07
                                                                                                                                     Year


      20




                                : A Computational Microscope
                                                                                                                                                              20,000
                                                                                  100
                                                                                                                                                            processors
     Four Highlighted Features                                                                           NAMD
                                         Simulation Rate in Nanoseconds Per Day




1.    Ground Breaking Simulations                                                      10                scales
2.    Active User Base                                                                                   by 103
                                                                                           1
3.    Extremely Scalable
4.    “Harnessing Parallelism for                                                   0.1
                                                                                                                                                            LYSOZYME (40K atoms)
                                                                                                                                                            APOA1 (92K atoms)
                                                                                                                                                            ATPase (327K atoms)
      Small and Fast Simulations”                                                                                                                           STMV (1M atoms)
                                                                                                                                                            BAR d. (1.3M atoms)
                                                                                                                                                            IAPP (5.5K atoms)
                                                                                  0.01
                                                                                                     1          10           100                1000         10000          100000
                                                                                                                                   Processors


                                                                                                         "We haven't found a hard limit
                                                                                                         on scaling up the number of
           NAMD scales by 103
                                                                                                         processors."
  IAPP                                                                                                                -- Philip Blood and Greg Voth,
                                                                                                                                  Univ Utah
                                                                                                            Commenting on NAMD performance
                                  STMV                                                                      for the PSC XT3 Cray
      21
                              : A Computational Microscope
                                            dihydrofolate reductase        DHFR Benchmark
                                                                                23,558 atoms
                                                                             2489 protein atoms
 Four Highlighted Features                                                   21,069 water atoms
                                                                                1 fs timestep
1. Ground Breaking                                                          62.23 Å3 System size
                                                                          64 x 64 x 64 PME Grid
   Simulations                                                             9 Å non-bonded cutoff
                                                                          NVE Ensemble, 300 K
                                                              MTX      Bonds to Hydrogen constrained
2. Active User Base                                                            92 atoms / core

3. Extremely Scalable
4. “Harnessing Parallelism
   for Small and Fast                                                           31 ns/day
                                                                            on 256 processors
   Simulations”



      22




      NAMD: Practical Supercomputing
  •   20,000 users can’t all be computer experts.
       – 18% are NIH-funded; many in other countries.
       – 4200 have downloaded more than one version.
  •   User experience is the same on all platforms.
       –   No change in input, output, or configuration files.
       –   Run any simulation on any number of processors.
       –   Automatically split patches and enable pencil PME.
       –   Precompiled binaries available when possible.
  •   Desktops and laptops – setup and testing
       – x86 and x86-64 Windows, PowerPC and x86 Macintosh
       – Allow both shared-memory and network-based parallelism.
  •   Linux clusters – affordable workhorses
       – x86, x86-64, and Itanium processors
       – Gigabit ethernet, Myrinet, InfiniBand, Quadrics, Altix, etc
          Simulation of the Entire Ribosome
      Collaborators: Joachim Frank (HHMI), Willy Wriggers (UT Houston)

                                                                                            100

                                                                                                       IAPP (islet amyloid peptide)

                                                                                             10




                                                   Simulation Rate in Nanoseconds Per Day
                                                                                                                                                     NAMD
                                                                                              1                                                      scales
                                                                                                                                                     by 103
                                                                                                                                 IAPP (5.5K atoms)

- 2.8 million-atom simulation of                                                             0.1                                 LYSOZYME(40K atoms)
                                                                                                                                 APOA1 (92K atoms)
ribosome with NAMD2.6 (Core 3)                                                                                                   ATPASE (327K atoms)
                                                                                                                                 STMV (1M atoms)
- 2,048 processors on a Cray XT3                                                                                                 BAR D (1.3M atoms)
                                                                                                                                 RIBOSOME (2.8M atoms)
- 3ns / day                                                                                 0.01

- Pencil decomposition for PME                                                                     1         10        100
                                                                                                                    Processors
                                                                                                                                      1000             10000


provided a 20% speedup
 11




      NAMD: Designed for Clusters
•     Resource has pioneered clusters for MD:                                                                      June 2001
       – 1993: HP workstation cluster, 12 nodes
            • Original NAMD development platform.
       – 1998: Pentium II Linux cluster, 16 nodes
       – 2001: Athlon Linux clusters, 3x32 nodes
            • Used for first cluster workshop in June.
            • On-demand queues for short jobs.
       – 2003: Dual-Athlon clusters, 3x24 nodes
       – 2004: Dual-Xeon clusters, 3x24 nodes
       – 2005-2006: Five hands-on workshops                                                                       April 2006
            • 24 students per workshop.
            • Students build clusters in teams of four.
•     Most popular platform in 2005 NAMD user survey was
      “Local Linux cluster.”
             NAMD Shines on InfiniBand
                       Fall 2006: TACC Lonestar comes online.
                         Commodity cluster with 5200 cores!
             100                                                     32 ns/day
                                                                    2.7 ms/step
                                                    m s)
                                            k   ato
              10                      R (24                                    15 ns/day
ns per day



                                   F
                             C /DH            ms
                                                )                             5.6 ms/step
                           JA              ato
                                        2k                  s)
                                    1 (9              a tom
               1                  oA
                                Ap                (1M
                                               V
                                           S TM
              0.1
                       4        8    16   32     64        128 256 512 1024
                                                cores




    Porting NAMD to Abe Cluster
• Build on experience from TACC Lonestar
             – Serious performance bugs in MVAPICH 0.9.8
                    • NAMD/Charm++ stresses MPI with “unexpected” messages
             – Fixed with 0.9.9 beta 2 at TACC
• On Abe use MPICH-VMI instead
             – Fast and stable for most users
             – Million-atom simulations have network errors
• Work in progress
             – Native InfiniBand ibverbs port of Charm++
             – Actually try MVAPICH 0.9.8 on Abe
             – Detailed performance debugging at limits of scaling
                   NAMD on NCSA Abe
              NCSA Abe is based on Dell servers and InfiniBand.
             Commodity cluster with 1200 blades and 9600 cores!
             100                                                   26 ns/day
                                                     ms)          3.3 ms/step
                                            k   ato
                                        (24
             10              H     FR     s)
ns per day




                         C/D         atom
                       JA        92k                   s)
                              1 (                a tom
                          ApoA               (1M                                 4.7 ns/day
              1                          MV                                      18 ms/step
                                      ST                                        7 cores/node


             0.1
                   8      16       32     64     128 256 512 1024 1792
                                                 cores




      NCSA Abe vs TACC Lonestar
              Both clusters based on Dell servers and InfiniBand.
             Difference is quad-core (Abe) vs dual-core (Lonestar).
             100                                                   32 ns/day
                                                     ms)          2.7 ms/step
                                            k   ato
                                 R      (24                                        15 ns/day
             10                HF                   s)                            5.6 ms/step
                                                 tom
ns per day




                             /D
                       JAC           2k
                                                a                   ms)
                                 1 (9                           ato
                               oA                        ( 1M
              1
                             Ap                       MV
                                                                                 4.7 ns/day
                                                ST                               18 ms/step
                                                                                7 cores/node

             0.1
                   4     8        16    32      64    128 256 512 1024 1792
                                                 cores
   NAMD projects run well on Abe
                                                                                        chromatophore
                                                           silica
      polio virus                                        nanopore
                  System                  Size (atoms)   CPUs   Benchmark(ns/day)   SUs used
                  Apolipoprotein A-1      98,000         96     3.9                 12,000
                  Discoidal HDL           144,000        128    3.6                 12,800
                  STMV                    1,080,000      384    0.9                 64,300
                  Lipase                  34,000         384    64.0                45,000
                  Silica Nanopore         46,000         32     2.9                 24,000
                  FD coat protein         57,000         192    13.0                10,000
                  Translocon SecY         106,000        128    5.0                 30,000
                  Chromatophore LH2       910,000        256    1.2                 49,100
                  Poliovirus              1,000,000      1024   2.5                 49,200
                  TOTAL                                                             296,400


       lipoprotein
                                                                                               hearing




                                       Molecular Basis of Hearing




                                                                            Inner Ear Mechanism



                                                                                    M. Sotomayor, D.
                                                                                    Corey, and K.
                                                                                    Schulten Structure,
                                                                                    13:669-682, 2005
Tip Links (Kachar       Hair bundle (Assad
et al., 2000; Corey     and Corey, from Sensory
Lab)                    Transduction, G. L. Fain).
                                            Molecular Basis of Hearing




                                                                                  Inner Ear Mechanism




Tip Links (Kachar           Hair bundle (Assad
et al., 2000; Corey         and Corey, from Sensory
Lab)                        Transduction, G. L. Fain).




         Molecular dynamics simulations of complete C-cadherin
         extracellular domain in water with and without calcium:
                             345,000 atoms
                                                     • Crystal Structure of C-cadherin from Boggon et
                                                     al. 2002 (pdb code 1L3W).
                                                     • >100 ns of combined simulation time using
                                                     multiple supercomputers.
                                                     • Abe: 0.38 days/ns (2.6 ns/day) using 344
                                                     processors.
                          +Ca2+                                                  -Ca2+




    •1L3W with five modules (without sugars) + 12                  •1L3W with five modules (without sugars) no
    calcium ions + 2 disulfide bonds on EC5 +                      calcium ions + 2 disulfide bonds on EC5 +
    Crystallographic Water                                         Crystallographic Water
             –345K atoms total, 12 Ca+, 8 Na+, 364x105x94Å               – 345K atoms total, 32 Na+, 364x105x94Å
    •Equilibration (NpT, PBC, PME, CMAP,dt=1,1,1)                  •Equilibration (NpT, PBC, PME, CMAP,dt=1,1,1)
          –Protein restrained (100ps)                                    – Protein restrained (100ps)
          –Protein released (1ns NpT + 4ns in NVE)                       – Protein released (1ns NpT + 4ns in NVE)
 Calcium Ions Protect Cadherin from Unfolding




Steered Molecular Dynamics (k=1kcal/mol/Å2; v=0.1 or 0.01Å/ps )




                  Biomedical Studies of Lipoproteins

                                      •   Residue-based coarse-graining developed to simulate
                                          nanodisc assembly & disassembly
                                      •   CG Builder: coarse-graining and reverse coarse-graining
                                      •   Compare simulation results with experimental SAXS
 How do nanodiscs self-assemble?      •   Current biomedical applications include experimental
                                          studies of membrane bound
                                           –    Cytochrome P450
                                           –    G-Protein Coupled Receptors
                                           –    Blood Clotting Tissue Factors
                                           –    High-Density Lipoproteins




                                                                                Collaboration with
                                                                                Steve Sligar




    Nanodisc Assembly              SAXS at APS, Argonne, IL
    32
                  High-density lipoproteins
                                                 •    Discoidal HDL: two apolipoprotein A-I
                                                      wrapped around a POPC bilayer
                                                       – 144,000 atom system equilibrated for
                                                          15 ns
                                                       – 3.6 ns per day using 128 NCSA Abe
                                                          CPU’s (16 nodes)




  •     Lipid-free apolipoprotein A-I
         – 98,000 atom system equilibrated for
           20 ns
         – 3.9 ns per day using 96 NCSA Abe
           CPU’s (12 nodes)




Studying the Physical Mechanism of Photosynthesis

                                                     Describe entire living
                                                        cells at atomic
light                                                     resolution




                               ADP         ATP
Studying the Physical Mechanism of Photosynthesis


                                          How does membrane
light                                         curvature
                                               develop?




                          ADP       ATP




         Protein Packing Induces Membrane
                     Curvature
   Seven Rb. sphaeroides peripheral light
   harvesting (LH2) complexes in mixed
   POPE/POPG membrane patch                  top view

        909830 atoms
        ~10 ns full equilibration

        ~ 1.2 ns/day on 32 Abe nodes
        NPT equilibration, Charmm
        force field
                                             side view
                 Mechanism of Poliovirus Entry
              Collaboration with J. Hogle and X. Zhuang (Harvard)

In vivo:
             virus,                     Swollen and
                       capsid
            “160S”                      partially open
             state       RNA            “135S” state                       “80S” state

receptors


     cell
                                                                    cell
                                 cell
                                                                               RNA release

In vitro:
                                                         135S
    160S              T ~> 37 C

                                                                               160S
     +                  low T




   Studying the Initial Transition in Poliovirus
      Capsid (160S " 135S): Structures
                                                                      All-atom model
Poliovirus+Receptors: cryo-EM
                                                                   (160S with receptors)




                           Homology model of the receptor fitted
                                into the cryo-EM density
         46 nm
Obtaining the 135S structure by fitting 160S into the 135S cryo-EM map:

                                   Real dynamics and
                                interactions in between?


                                   “Abe” to the rescue!
            Initial                                                        After the fitting
  Studying the Initial Transition in Poliovirus
 Capsid (160S " 135S): Dynamics on “Abe”
                                                 System size: 1,000,000 atoms

                                                 Simulated time: 5 ns (and counting)

                                                 Processors: 1024 (128 nodes)

                                                 Performance: 2.5 ns/day

                                                 Simulation detail: NpT ensemble,
                                                 CHARMM forcefield, NAMD


                                                Results of this simulations, in
                                              conjunction with the160S"135S
                                                fitting, will be used to identify
                                              the capsid-receptor interactions
Segment of the capsid                           responsible for the transition
  with 5 receptors




              Satellite tobacco mosaic virus (STMV)

STMV - 60 protein icosahedron around an RNA
   core
• 1,079,000 atom system
• Six 1-ns pulling simulations performed on
   abe
• 0.82 ns per day using 384 NCSA Abe CPUs
   (48 nodes)




                                     •   Steered molecular dynamics (SMD) applied to
                                         study stabilizing interactions
                                     •   SMD at different protonation states used to probe
                                         pH-driven particle disassembly
                                     •   Results so far highlight importance of protein-
                                         RNA interactions in the protein’s
                                         N-terminal “tail”
                                     •   Ongoing simulations on abe to further
                                         characterize these interactions
    DNA Recording with Synthetic Nanopores
     Electronic decoding of DNA sequence using synthetic
     nanopores
     A low cost, rapid DNA sequencing method would significantly
     impact life science and personal medicine
     Integrates both biomolecular and inorganic structures via methods
     extensible to more complex systems




                         Wong & Pettitt,
                         Biopolymers 73: 570-578 (2004)
                         an early treatment!




Current rectification in silica nanopores
                                                          •   Asymmetric current-voltage curves (so-
                                                              called current rectification) are observed in
                                                              nanopores.
                                                          •   The reason of the asymmetry was not clear.
                                                              No experimental technique can visualize the
                                                              ion dynamics inside the nanopore.
                                                          •   MD simulations act as microscopes, reveling
                                                              that current rectification is due to ion
                                                              adsorption on the nanopore surface

                   •   System size: 46000 atoms.
                   •   8 different MD runs of 10 ns                                           Ideal
                       each.
                   •   2.7 ns/day using 32 NCSA                                                  Simulation
                       Abe CPU’s (4 nodes)                          Current rectification
       NCSA Abe user experiences
• Positive
   – NAMD scales well
       106K atoms: 128 CPU (100%), 256 CPU (81%), 512 CPU(73%)
   – Fast computing
   – Short queues (large jobs run quickly!)
• Negative
   – Multiple crashes upon submission experienced by large jobs
   – Jobs that have “died” continue to be listed as running even though no
     output is being generated
   – Hardware failure resulting in inaccessible scratch space

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:11/13/2011
language:English
pages:17