Development and Application of
Parallel Plume-in-Grid Models
Prakash Karamchandani, Krish Vijayaraghavan, Shu-Yun
Chen and Christian Seigneur
AER, San Ramon, CA
7th Annual CMAS Conference
October 6–8, 2008
Chapel Hill, NC
Limitations of Traditional Grid
Modeling
• Horizontal resolution of a few kilometers to tens of
kilometers cannot resolve sub-grid scale effects such as
transport and chemistry of point source emissions
Plume Size vs Grid Size (from
Godowitch, 2004)
• Artificial dilution of stack
emissions
• Unrealistic near-stack
plume concentrations
• Incorrect representation of
plume chemistry
• Incorrect representation of
plume transport
Plume Chemistry & Relevance to
Ozone and PM Modeling
3
2
Early Plume Long-range Plume
Dispersion Mid-range Plume Dispersion
Dispersion
1 NO/NO2/O3 chemistry Reduced VOC/NOx/O3
chemistry — Full VOC/NOx/O3
acid formation from OH chemistry —
and NO3/N2O5 chemistry acid and O3 formation
Plume-in-Grid (PiG) Modeling
• Plume-in-Grid (PiG) approach provides a sub-grid
scale representation of stack plumes
• Addresses inability of 3-D grid models to correctly
simulate atmospheric fate of stack emissions
• Approach consists of embedding a reactive puff
model within the grid model
• The initial transport and chemistry of point source
emissions are treated with the puff model
• When puff sizes are comparable to grid model
resolution, the puffs are merged with the grid and
subsequent calculations are done with the grid
model
AMSTERDAM
• Advanced Modeling System for Transport, Emissions,
Reactions & Deposition of Atmospheric Matter
• Suite of models based on CMAQ v 4.6, October 2006
release and with alternate options
• Options:
– MADRID treatment for PM: Model of Aerosol Dynamics,
Reaction, Ionization and Dissolution
– APT: Advanced Plume Treatment with embedded
plume model SCICHEM (state-of-the science treatment
of stack plumes at the sub-grid scale)
• APT can be used with either the MADRID treatment for
PM or the CMAQ treatment (currently available for AERO3
option)
AMSTERDAM Components
CMAQ v. 4.6
MADRID PM Treatment
CMAQ-MADRID
SCICHEM-AERO3 SCICHEM-MADRID
PM Treatment based on EPA CMAQ PM Treatment based on CMAQ-MADRID
CMAQ-AERO3-APT CMAQ-MADRID-APT
SCICHEM
• Three-dimensional puff-based model
• Second-order closure approach for plume
dispersion
• Puff splitting and merging
• Treatment of plume overlaps
• Optional treatment of building downwash
• Optional treatment of turbulent chemistry
• PM, gas-phase and aqueous-phase chemistry
treatments consistent with host model
Model Applications/Evaluations
• North-eastern U.S. with two nested grid (12 km
& 4 km) domains (NARSTO)
– 5 day episode (O3 only)
– 30 point sources simulated explicitly
– PiG model about 1.5 times slower than grid
model
• Central California (4 km resolution)
– 3 day episode (O3 only)
– 10 point sources simulated explicitly
– PiG model about 1.2 times slower than grid
model
Model Applications/Evaluations
(continued)
• Eastern U.S. (VISTAS 12 km domain)
– 2 month simulations for O3 and PM
– 14 point sources simulated explicitly
– PiG model about 1.25 times slower than grid
model
• Southeastern U.S. (ALGA 12 km domain)
– Annual simulations
– O3, PM, Hg and nitrogen deposition
– 40 point sources simulated explicitly
– PiG model about 1.8 times slower than grid
model
Typical Results
• The PiG treatment has a strong effect on model
predictions of surface O3 titration (near large NOx
point sources), as well as O3, sulfate and nitrate
formation downwind of large NOx and SO2 point
sources
• A purely gridded approach typically
overestimates PM production downwind of large
NOx point sources because it overestimates SO2
to sulfate and NOx to nitrate conversion rates
near the stack
• Overall model performance statistics are almost
identical between the grid-only and PiG
treatments, but observed plume events are better
captured in the PiG approach than in the purely
gridded approach
Power-Plant Contributions to 24-hr
Average Sulfate Concentrations
CMAQ-MADRID CMAQ-MADRID-APT
Change in Power-Plant Contributions (%) to
PM2.5 Sulfate Concentrations When a PiG
Approach is Used
Conversion of Power Plant
SO2 Emissions
Domain-wide mass-budget analysis performed for SO2
and sulfate attributable to power plant emissions
Sulfate to Total Sulfur Ratios (%)
Emissions CMAQ-MADRID CMAQ-MADRID-APT
January 2.25 17.8 15.6
July 2.35 75.5 67.4
Approximate SO2 Conversion (%)
CMAQ-MADRID CMAQ-MADRID-APT Change
January 15.9 13.7 -14%
July 74.9 66.6 -11%
Spatial Distribution of
Total Nitrogen Deposition
Change in annual dry + wet deposition flux due to power plant NOx controls
CMAQ-MADRID CMAQ-MADRID-APT
Maximum
reduction in 0.85 kg/ha 0.42 kg/ha
deposition flux
APT: Less oxidation of NOx to HNO3 => Less dry deposition near the plant
PiG Modeling Constraints
• Can be computationally expensive if a large number
of point sources are treated with the puff model –
computational requirements increase by a factor of
two to three for 50 to 100 sources
• Point sources have to be selected carefully to limit
the number of sources treated
• To obtain results in a reasonable amount of time,
annual simulations are usually conducted by
dividing the calendar year into quarters and
simulating each quarter on different processors or
machines
• Parallel version of code can address these
constraints
Parallelization of APT
• Parallelization of stand-alone version of puff
model (SCICHEM)
• Adaptation of parallel stand-alone SCICHEM to
parallel plume-in-grid version
• Development of appropriate parallel interfaces
between parallel CMAQ-MADRID/CMAQ-AERO3
and parallel plume-in-grid version of SCICHEM
Parallelization of Host Models
• CMAQ and CMAQ-
MADRID
– Based on
Message
Passing
Interface (MPI)
– Horizontal
domain
decomposition
Parallelization of SCICHEM
• SCICHEM
– Puff decomposition
for chemistry step
– Needs access to
entire 3-D grid
– Uses MPI
Parallel Interface
• Gathers 3-D sub-domain concentrations from
the various processors to create a global 3-D
concentration array
• Provides global 3-D concentration array to
SCICHEM
• After the SCICHEM time step, the modified
global 3-D concentration array is scattered to
the 3-D sub-domain concentration arrays
Model Interaction Diagram
Domain, grid information
Geophysical data
Meteorological data
Deposition velocities
Control
Emissions, I/O File
IC/BC API
I/O
Parallel API Point
Parallel Parallel
CMAQ-MADRID/ source
I/O Interface SCICHEM
CMAQ-AERO3 emissions
API
I/O
Output API Output Standard
Merge SCICHEM
concentrations, puff
puffs output
Deposition, information
Puff diagnostics
Puff diagnostics
Current Status
• Development of parallel version of AMSTERDAM
completed in 2008
• On a 4-processor machine, the parallel version is
about 2.5 times faster than the single-processor
version
• On-going project to apply the model to the central
and eastern United States at 12 km resolution and to
evaluate it with available data
– Over 150 point sources explicitly treated with APT
– Annual actual and typical simulations for 2002
– Future year emission scenarios
– Other emission sensitivity scenarios
Ongoing Application of Parallel
PiG Model
• 12 km grid resolution
• 243 x 246 x 19 grid cells
• Over 150 PiG sources
Acknowledgements
• Funding for model development, including
parallelization, and model application:
– EPRI
• Collaboration on parallelization:
– L-3 Communications Titan Group
• Parallelization insights:
– Dr. David Wong, EPA