Global Atmosphere 7.1 N216


To compare the performance on our systems with the MetOffice results, we have created a local version of the MetOffice's GA7.1 N216 model, specifically of their High Resolution Model Intercomparison Project. The ID of this rose is u-an493, which is a copy of the MetOffice's u-ai718, with only very few changes to make it compatible with NCI raijin:

  • The paths to the ancillary files were adjusted
  • The number of CPUs was changed to 26x32 (with 2OMP threads)
  • The submission scripts were adapted for NCI (PBS, modules, etc)

Performance


  • There are two recommended decompositions:
    • By default, the job runs on 24 nodes (16 cores each), specifically 24x16, with no OpenMP. This is a reasonable compromise between speed (~28 hr/yr) and cost (~11kSU/yr)
    • The fastest decomposition is 26x32, with 2 OMP threads, or 104 nodes. Adding more CPUs only increases overhead. It can model a year in about 11 hours, but at a cost of 20kSU per model year.
  • The resubmission frequency is set to 1 month, and should not be changed to less than 10 days to not impact performance.

Obtaining the Job


As the rose is located on the MetOffice Collaboration Repository, you need an account there [link missing].

$ mosrs-auth
$ access-auth
$ rosie copy u-an493

Notes



External Links


http://collab.metoffice.gov.uk/twiki/bin/view/Project/HiResCL/HiResCLPRIMSimulations
https://code.metoffice.gov.uk/trac/ukcmip6/wiki/CMIP6HighResMIP
https://code.metoffice.gov.uk/trac/ukcmip6/wiki/CMIP6HighResMIP/highresSST-present/HadGEM3-GC31-N216-ORCA025
https://code.metoffice.gov.uk/trac/ukcmip6/wiki/CMIP6HighResMIPForcings
https://code.metoffice.gov.uk/trac/roses-u/log/a/n/4/9/3/trunk

Global Atmosphere 7.1 N96


A version of HighResMIP is also available at N96 resolution, using the job u-as709.

As in the n216 version, only minor changes were made to the model's data paths (Local copies of the files used at the Met Office can be found under /g/data1/w35/saw562/HighResMip/data)

On Raijin's 'normal' queue with 16x16 processors this costs ~ 600 SU per model year, and takes ~ 6 hours per model year to run.

Transpose AMIP style N216 runs


Transpose AMIP style hindcast runs can be run using the job u-ap508. This allows researchers to set the model start date to an arbitrary time and focus on specific weather and climate events.

This configuration is based on the HighResMIP configuration, but uses a Gregorian calendar rather than the original 360 day calendar.

Initialised fields


The following fields are sourced from the ERA Interim GRIB datasets under /g/data1/ub4. Details of how this works can be found in UMDP 303

ECMWF short name
Description
z
Geopotential height
lsm
Land sea mask
sp
Surface pressure
skt
Skin temperature
sd
Snow depth
stl1
Soil temperature by layer
stl2
stl3
stl4
u
U wind
v
V wind
t
Temperature
q
Specific humidity

Additionally a number of model fields required by the model are initialised to zero, and require some spin-up time

STASH code
Description
0155
Accumulated surface runoff
0156
Accumulated sub-surface runoff
0157
Gridbox area
0290
Daily accumulated lake flux
0511
Inland basin flow

Updated Fields


Sea surface temperature and sea ice are updated from ERA-Interim at 6 hourly intervals. The ancillary files use for this are generated by the `install_ancil` task, they are regridded to the model resolution using NCO then converted to UM format with mkancil.

The `install_ancil` task also sets up the calendar for EasyAerosol input files. As these files are large (the total size is ~1TB) the script first extracts just the dates important to the run before changing the calendar.

External Links

UMDP 303: Reading GRIB data using the UM
Roff et al. 2012 - Initializing ACCESS from Era-interim data