Computational Science Group @ CGAM
High Resolution Earth System
Modelling
Loïs Steenman-Clark
UK/Japan Workshop on Earth System
Modelling, October 2003
High resolution Earth System modelling is not only a
scientific but also a computational challenge.
Earth System modelling is like squeezing a quart into a pint pot
No problem if you know what a quart is and the size of a pint
No problem in a parallel world
1 quart = 2 pints
To squeeze a high resolution experiment into a supercomputer
we need to understand the supercomputer, the model as well as
the experiment to be run.
UK/Japan Workshop on Earth System
Modelling, October 2003
Supercomputer
UK/Japan Workshop on Earth System
Modelling, October 2003
Earth Simulator
Yokohama Japan
NEC, 640 nodes each with 8 processors
World number 1 in the TOP 500 list
HPCX
Daresbury, UK
IBM, 32 nodes each with 32 processors
Number 12 in the TOP 500 list
UK/Japan Workshop on Earth System
Modelling, October 2003
Use a kiviat diagram to visualise interdependencies
For the supercomputers the spokes
should be processor
performance
interconnect
I/O performance memory
Each spoke of the kiviat
diagram should depict
quantitative information HPCX, UK
UK/Japan Workshop on Earth System
Modelling, October 2003
Consider the spoke for processor performance
The scale could be in terms of several different quantities
Processor ES HPCX
Peak Tflops 40 6.6
Sustained (on Linpack) 35 3.2
Bandwidth to memory / vector 3 level
processor speed cache
Scales used on the spokes can be tuned to the best advantage.
It is difficult to provide a universal metric.
UK/Japan Workshop on Earth System
Modelling, October 2003
From a user perspective the supercomputer spokes
on the kiviat diagram should be
Availability could be
Availability • queue management
• fair share scheduling
ease of software • allocation provided
use • mean time between failure
cost
Earth Simulator HPCX
ES job scheduler Loadleveller with
(to be replaced) capability incentives Spokes on a user kiviat
diagram for a super-
Allocation through Allocation from computer are highly
collaboration NERC subjective.
MTBF=? MTBF99.5%
UK/Japan Workshop on Earth System
Modelling, October 2003
Model
UK/Japan Workshop on Earth System
Modelling, October 2003
Kiviat diagrams for each model in the Earth System
should be able to characterise critical dependencies
computation deduce model characteristics
such as scalability
communication
memory
I/O
Model HRES
• Unified Model
• Old dynamics
SRES
• Atmosphere
• Version 4.5
UK/Japan Workshop on Earth System
Modelling, October 2003
Earth Simulator
Initial high resolution N144 UM (vn 4.5) atmosphere run
for the intercomparison of diagnostics with the
T106 CCSR/NIES/FRSGC (vn 5.7b) model
SRES MRES GRES URES
HRES
N48 N96 N144 N216 N312
3.35 x 2.4 1.88 x 1.24 1.25 x 0.83 0.83 x 0.56 0.58 x 0.3
270 km 135 km 90 km 60 km 30 km
UK/Japan Workshop on Earth System
Modelling, October 2003
UK/Japan Workshop on Earth System
Modelling, October 2003
Experiment
UK/Japan Workshop on Earth System
Modelling, October 2003
The aim is to run a broad range of Earth System Model
experiments using the UM with
High horizontal resolution
Extended vertical resolution
Multi-component coupled experiments
(atmosphere, chemistry, ocean, land surface, sea ice)
as well as separate component assessment experiments
Multi-century runs with limited temporal resolution
or short experiments with high temporal resolution
Ensemble experiments
Currently, both on the Earth Simulator and HPCX, Earth System
modelling experiments using the UM are under development.
UK/Japan Workshop on Earth System
Modelling, October 2003
High Resolution Experiments
horizontal Kiviat diagram will be
vertical resolution different for each
resolution complexity particular experiment
ESM components
ensemble UM atmosphere diagnostics
size 5 daily mean fields
diagnostics
Mbytes
Issues 40
• rate of data output wrt experiment speed. 30
• long term storage wrt cost of re-running.
N144
• large data set analysis 20
make it a component of the Earth System 10 N96
Model.
N48
UK/Japan Workshop on Earth System
Modelling, October 2003
High Resolution Coupled Atmosphere-Ocean
experiments
Earth Simulator (Takahiro Inoue – RIST)
• scaling difficult due to => put the model on a memory
memory requirements diet
• load imbalance in models => improve component model
and between ESM components performance
• output data too huge => reduce diagnostics?
HPCX (Computational Science Group – CGAM)
• scaling poor due to slow => improve communications in the models
interconnect (interconnect upgrade in Spring 2004)
• load imbalance in models => improve performance and I/O strategy
• output is a challenge => devise new diagnostic strategies
UK/Japan Workshop on Earth System
Modelling, October 2003
Conclusions
High resolution Earth System modelling is a challenge
• need flexible and generic technical solutions that can be applied to all
Earth System Model components and all types of experiments.
• accept that future experiments with current models will not be
super efficient as manpower and expertise required to radically
change today’s models is too costly and can be best used elsewhere.
• seek to capture and pool expertise from running current Earth
System model experiments on today’s supercomputers to formulate
new strategies and computational techniques for the next generation
Earth System models.
UK/Japan Workshop on Earth System
Modelling, October 2003