The Evolving Environment at NCCS
Compute Node Linux
• Jaguar will be down November 1-5 to change from
Catamount to Compute Node Linux (CNL).
• CNL is a stripped-down version of Linux on the compute
• Try the system running CNL that is accessible to all
jaguar users: jaguarcnl.ccs.ornl.gov.
• Jaguarcnl contains 2,976 dual core XT3 compute nodes
each running CNL.
• Eight additional cabinets will be removed from jaguar
and added to jaguarcnl increasing it from 32 cabinets to
40 cabinets (7440 cores).
Changes for Compute Node Linux
• Currently only the /tmp/work/$USER lustre directory is
visible to compute nodes.
• There are several important differences between
Catamount and CNL, and users are encouraged to view
the details of those differences at
• Due to the differences between CNL and Catamount,
users should recompile on the CNL system prior to
• The batch option “-lsize” specifies cores under CNL.
• aprun replaces yod as the job launcher under CNL.
• Each user is currently allowed only one job in a running
• The maximum walltime limit is 12 hours.
CNL batch example
#PBS -A STF006
#PBS -N test
#PBS -j oe
#PBS -q debug
#PBS -l walltime=0:50:00,size=4
aprun -n4 ./cqlp
· The pathscale compiler is now available on jaguar and
· Before compiling:
- module swap PrgEnv-pgi PrgEnv-pathscale
· Use ftn –O3 instead of ftn –fast for compiler optimization
· Use -Wl,--allow-multiple-definition for the ldr flag if you get
too many warnings about multiple definitions
· Use a SuperLU library compiled with pathscale:
• Jaguar shutdown for upgrade: December 4, 2007- ?
• During this time users will have access to the remaining
40 XT cabinet transition system (7440 cores).
• Jaguar will be upgraded to 250 teraflops.
• The dual-core processors will be replaced with quad-
• Memory will be doubled.
• This transition will involve significant down time.
• Consider using OpenMP across the cores of a processor
• A petaflop system will be installed at the NCCS in 2008.
• The LCF Petaflop system will also be quad-core.
• It will come with a new interconnect that should enable
better scalability than the current XT4.
• This new interconnect is called Gemini and it will allow
you to effectively perform Co-Array Fortran, UPC and