FLASH: Overview and Status
Paul Ricker
University of Illinois
FLASH 0 - 2.5
K. Antypas, A. Caceres, A. C. Calder, L. J. Dursi, B. Fryxell, B. Gallagher, T. Linde,
K. Olson, T. Plewa, K. Riley, D. Sheeler, A. Siegel, F. Timmes, N. Vladimirova,
G. Weirs, M. Zingale
FLASH 3.x
P. Brune, A. C. Calder, A. Dubey, R. Fisher, B. Gallagher, C. Graziani, C. Jordan,
D. Lee, L. Reid, P. Rich, K. Riley, D. Sheeler, N. Taylor, D. Townsley, K. Weide
University of Illinois/NCSA
R. Balakrishnan, Z. Lukić, P. M. Sutter, C.-C. Yang, H.-Y. Yang
Durham Radiation Transfer Workshop, September 2007 1
FLASH
Adaptive mesh refinement (AMR) simulation
code originally developed under DOE ASCI
program at U. Chicago
Original science targets
Type Ia supernovae, novae, X-ray bursts
Code targets
Performance and scalability
Modularity and ease of use
Verification and validation
Portability
Publicly available and used for a variety
of problems
> 300 users
> 170 refereed publications
A. Calder (UC/SUNY)
Durham Radiation Transfer Workshop, September 2007 2
FLASH history
FLASH 0 (1998) FLASH 1.x (1999-2000)
FLASH 2.x (2001-2005) FLASH 3.x (2006-)
Durham Radiation Transfer Workshop, September 2007 3
User community (fall 2005 survey)
Breakdown of FLASH code research
areas for primary research tool users
Over 250 refereed papers published
using the FLASH code
Durham Radiation Transfer Workshop, September 2007 4
Framework
PARAMESH
Grid
Uniform
PPM
Hydrodynamics MHD
Driver Kurganov
Diffusion Multigrid
Initialization
Radiative cooling
Evolution Source terms Nuclear burning
Flame model
Mesh database Particle-mesh
Runtime Particles
parameters Tracers
Log file
Performance Conductivity
monitor
Materials
Eqn of state
Physical constants
Multipole
Gravity
Multigrid
HDF 5
I/O
NetCDF
Component interface layer FLASH 2 – Ricker et al. (2000)
Durham Radiation Transfer Workshop, September 2007 5
Automated testing
FlashTest runs a variety of test problems daily to verify the code on
different platforms
Web-based interface allows easy access to test results
Essential for immediately identifying bugs introduced during
development
Also tracks daily code performance changes
Test framework available for separate download
Green light indicates all
Platform runs were successful
Date of run
Floating statistics box gives
immediate overview of results
Red light indicates 1 or more
tests failed
Durham Radiation Transfer Workshop, September 2007 6
Adaptive mesh – PARAMESH library (MacNeice et al. 2000)
Simplified approach due to Quirk
(1991) and De Zeeuw & Powell (1993)
Each block contains nd zones in d
dimensions
Blocks stored in 2d-tree data structure
Factor of 2 refinement per level
Blocks mapped to space-filling curve
block
interior
zone boundary
Durham Radiation Transfer Workshop, September 2007 7
Adaptive mesh – load balancing
23 24 31
19 32 36 37
22
15 30
18 2021 28 35
16 17 29 33 34
14 27
12 13 26
25
11
4 5
9 10
3 8
1
2 6 7
1–9 10 – 18 19 – 27 28 – 37
Fryxell et al. (2000)
Proc 0 Proc 1 Proc 2 Proc 3
Durham Radiation Transfer Workshop, September 2007 8
Physics: hydrodynamics
Piecewise-Parabolic Method (PPM; Colella & Woodward 1984)
Cartesian and curvilinear coordinates
General convex equation of state (Colella & Glaz 1989)
Multiple fluids (both tracers and physical fluid components)
Explicit diffusion and viscosity terms
Rotating reference frames
Mesh refinement strategy
Second derivative (Löhner 1987)
Fryxell et al.
(2000)
Durham Radiation Transfer Workshop, September 2007 9
Physics: magnetohydrodynamics
Currently implemented: directionally split
Powell et al. (1999) 8-wave solver
Ideal, resistive, Hall MHD
Diffusive and elliptic projection ∇⋅B cleaning
Under development: unsplit Godunov solver
Orszag-Tang vortex
Ideal MHD
Constrained transport – identically preserves
∇⋅B = 0
Dissipation control algorithm
2D version ported to FLASH 3, 3D version
being developed
Magnetized bubble
(Robinson et al. 2003)
Durham Radiation Transfer Workshop, September 2007 10
Physics: equation of state and reactions
Timmes (1999)
Equations of state / fluid compositions
Perfect fluid and mixture of perfect
fluids
Stellar EOS with 7, 9, 13, 19 isotopes
Hydrogen, helium, plus metals
Solar corona composition
Reaction terms / networks
Nuclear reactions (several networks,
including some with NSE treatment)
Arrhenius and KPP burning rates
Metallicity-dependent radiative cooling
(collisional ionization equilibrium)
Nonequilibrium and equilibrium
ionization
Heating and stirring models
Durham Radiation Transfer Workshop, September 2007 11
Physics: particle transport
“Active” particles
Contribute to and feel forces
Mesh mapping: NGP, CIC, TSC, CIC in
cylindrical or spherical geometry
Time integration: Euler, leapfrog
Models: dark matter, stars, DSMC
“Passive” particles
Tracers of gas velocity field (Marker Particle
technique; Rider & Kothe 1995)
Initial distribution: uniform, by density
Mesh mapping: quadratic
Time integration: predictor-corrector
Durham Radiation Transfer Workshop, September 2007 12
Physics: gravity
External fields
Point source
Plane-parallel
Self-gravity
Multipole solver – isolated problems
Multigrid solvers – isolated and periodic
problems
● Old solver – weighted Jacobi relaxation
● New solver – local FFTs
Parallel FFT
● FFTW (slabs) – T. Theuns
● PFFT (pencils) – A. Dubey
Treecode
Durham Radiation Transfer Workshop, September 2007 13
Physics: gravity
Huang & Greengard (2000); Ricker (2006)
Ideas
“Black box” direct solver (FFT) with Dirichlet
boundaries on each block
Residual yields single-layer potential correction
Benefits
Fewer boundary-value transfers needed
Immediate zeroing of residual in block interiors
Durham Radiation Transfer Workshop, September 2007 14
Physics: cosmology
Comoving coordinates
Scale factor evolution: + matter,
or tabulated a(t)
Initial conditions
Read directly from multi-level
GRAFIC output
Additional physics
Scalar field dark energy
CDM – WMAP3 256 h-1 Mpc
Durham Radiation Transfer Workshop, September 2007 15
Physics: cosmology
Refinement methods
Logarithmic density thresholds
Number of particles per zone
Resimulation – particle masks
Adaptive particle smoothing
If mesh refines but particle mass
stays the same, can get artificial Level
accretion 2
Solution: for each particle, choose
mapping level based on local density
Also fixes momentum
nonconservation for particle clouds
1
overlapping refinement boundaries
Durham Radiation Transfer Workshop, September 2007 16
Physics: subgrid models
Validation / basic physics
Turbulent dissipation
Supernovae
Deflagration fronts
ISM / galactic structure
Formation and evolution of
individual stars
Cosmology
Formation and evolution of
populations of stars
Durham Radiation Transfer Workshop, September 2007 17
Performance – turbulence (weak scaling)
LLNL BlueGene/L, early 2006
57 sec/cell/step
643 zones/proc
83 particles/proc
A. Dubey (U. Chicago)
Durham Radiation Transfer Workshop, September 2007 18
Performance – cosmology (strong scaling)
Constant total work – 1283 uniform grid / 1283 particles (July 2005)
Old multigrid Poisson solver; old block redistribution algorithm
Durham Radiation Transfer Workshop, September 2007 19
Getting the code
Visit http://flash.uchicago.edu
Fill out web form
Print, sign and fax user agreement to the Flash Center
Receive download URL by email
What you will need
Python
Fortran 90 and C compilers
MPI
HDF5 (if you plan to write any output...)
Durham Radiation Transfer Workshop, September 2007 20
Using the code
Easy!
Create your problem setup
● Make a directory for it under setups/ (FLASH 3:
source/Simulation/SimulationMain/)
● Copy and modify an existing init_block.F90 (FLASH 3:
Simulation_initBlock.F90)
Create Modules (FLASH 3: Units) file containing list of modules
(FLASH 3: units) to include
./setup your_problem -options
make
Create a runtime parameter file (flash.par)
mpirun -np X ./flash3
Sources of help
Read the manual!!
What does routine X do? - RoboDoc online documentation
Subscribe to flash-users
Durham Radiation Transfer Workshop, September 2007 21
Future of FLASH
Which version to use?
FLASH 2.5 – most feature-complete, but in maintenance mode
FLASH 3 – in public beta, most ported modules nearing completion
Continued maintenance
Flash Center has applied to ASCI followon program, PSAAP
DOE, U. Chicago, and Argonne have all expressed an interest in
keeping FLASH supported
Upgrades and new modules
User community continues to produce new modules – need a
mechanism to make them available (“FlashForge”)
U. Illinois, U. Chicago, and Argonne have put in a proposal to NSF
PetaApps solicitation for petascale enhancements
Seeking other sources of funding for continued development
Durham Radiation Transfer Workshop, September 2007 22