This version is from 2017. Look for the latest version available at NCI at this page.

Foreword


This document is intended to serve as a handy reference for new users in building and running the WRF and WPS on the RAIJIN cluster at NCI without the need of making site-specific modifications to WRF and WPS.
This document is only valid for version 3.9 of WRF and later. Please refer to the older version pages for other instructions.

Source Code


NCAR has moved the WRF source code to Git version control. The original code for WRF v3.9 can be found on NCAR's release Github repository.
The source code ported to Raijin can be found both under /projects/WRF/WRFV_3.9 and on Github: https://github.com/coecms/WRF

Clone the WRF Github repository (using HTTPS or SSH protocol depending on your setup) to your own space under /short:
cd /short/$PROJECT/$USER/
git clone -b V3.9 https://github.com/coecms/WRF.git
or
cd /short/$PROJECT/$USER/
git clone -b V3.9 git@github.com:coecms/WRF.git
 

Building WRF (ARW)


By default, WRF will output files in netcdf 3 format. If you would like to directly output files in netcdf 4 format with compression, you need to edit the file WRF/build.env before compilation. Note if you are planning to generate very frequent outputs, this will significantly slow down your simulation. You could be better off compressing the outputs afterwards but please don't forget to compress your data.

Go to the WRFV3/ subdirectory, run:
cd /short/$PROJECT/$USER/WRF/WRFV3/
There is a run_compile script to help you configure and compile WRF. This script by default will compile the WRFV3 software for a real experiment, with speed optimisations for the normal and express queues and with basic nesting. There is a range of command-line arguments accepted by the script. These are the command-line arguments accepted by the configure script, an option to clean everything prior to compilation, the choice of compilation flags and nesting type for the configure script and finally the compilation option for the compile script. You can access a full list and description by running:
./run_compile -h
To configure and compile WRFV3 with the default options, you simply run:
./run_compile
The configure step will be run interactively with the output on your screen, then the compilation step will be submitted to the express queue automatically. The compilation takes about 30-45 minutes on the express queue. To check the status, you can use the nqstat command.

After a successful build, one should see the following executable files for WRF (the ARW core):
[abc123@raijin3 WRFV3]$ ls -l main/*.exe
 
-rwx------ 1 abc123 wrf 35559897 Aug 22 17:14 main/ndown.exe
-rwx------ 1 abc123 wrf 35471615 Aug 22 17:14 main/real.exe
-rwx------ 1 abc123 wrf 35081246 Aug 22 17:14 main/tc.exe
-rwx------ 1 abc123 wrf 38763058 Aug 22 17:13 main/wrf.exe

Tip: if you are only running an idealised case and you are getting tired of specifying the option on the command-line, you can:
  1. change the default value of compile_case at the start of the run_compile script OR
  2. create an alias for the command in your .bashrc or .cshrc file (it is a good idea to change the name of the command for the alias so it is clear it is an alias if you need help later on):
bash syntax:
alias my_compile='run_compile --compile_case <replace with the compile option here>'
csh/tcsh syntax:
alias my_compile 'run_compile --compile_case <replace with the compile option here>'

Tip: run_compile is sourcing the environment as defined in WRF/build.env. Feel free to change the versions of software you want to use in here. Note the module purge at the start. This means your currently loaded environment will be completely ignored.

Debugging with WRF


If you want to compile WRF for debugging, the default option for compilation is not valid because it includes some optimisation. You'll need to use one of the first four architecture options. For example, to debug with the distributed memory only option (dmpar):
./run_compile -c -a 3 -D

Building WPS


Go to the WPS/ subdirectory, run:
cd /short/$PROJECT/$USER/WRF/WPS/
There is another run_compile script to help with configuring and compiling WPS. You can again choose the input to the configure script from the command-line option as well as cleaning a previous compilation of WPS. The default is to compile with GRIB2 support and a distributed memory code. See the inline help for run_compile for a full explanation:
./run_compile -h


To configure and compile WPS with the default options, you simply run:
./run_compile
A file configure.wps should be created after configuring has completed.

After a successful build, one should see the following executable files:
[abc123@raijin3 WPS]$ ls -l *.exe
 
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:19 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx 1 abc123 wrf 23 Aug 23 10:20 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx 1 abc123 wrf 21 Aug 23 10:20 ungrib.exe -> ungrib/src/ungrib.exe
[abc123@raijin3 WPS]$ ls -l */src/*.exe
 
-rwx------ 1 abc123 wrf 3412782 Aug 23 10:19 geogrid/src/geogrid.exe
-rwx------ 1 abc123 wrf 3206974 Aug 23 10:20 metgrid/src/metgrid.exe
-rwx------ 1 abc123 wrf 1209622 Aug 23 10:20 ungrib/src/g1print.exe
-rwx------ 1 abc123 wrf 1407507 Aug 23 10:20 ungrib/src/g2print.exe
-rwx------ 1 abc123 wrf 2169010 Aug 23 10:20 ungrib/src/ungrib.exe
-rwx------ 1 abc123 wrf 1184660 Aug 23 10:20 util/src/avg_tsfc.exe
-rwx------ 1 abc123 wrf 1250303 Aug 23 10:20 util/src/calc_ecmwf_p.exe
-rwx------ 1 abc123 wrf 1217842 Aug 23 10:20 util/src/height_ukmo.exe
-rwx------ 1 abc123 wrf 953821 Aug 23 10:20 util/src/int2nc.exe
-rwx------ 1 abc123 wrf 1118699 Aug 23 10:20 util/src/mod_levs.exe
-rwx------ 1 abc123 wrf 3704627 Aug 23 10:20 util/src/plotfmt.exe
-rwx------ 1 abc123 wrf 3409302 Aug 23 10:20 util/src/plotgrids.exe
-rwx------ 1 abc123 wrf 916613 Aug 23 10:20 util/src/rd_intermediate.exe
Note that there are some warnings in the build output like "warning: overriding commands for target `.c.o'." These are harmless and can be ignored.

Running WRF


Please follow the WRF tutorial to learn how to run WRF. For convenience, there are 2 run scripts under WRFV3/run:
  • run_real
  • run_mpi
These can be used to submit both real.exe and wrf.exe to the queues.

Tests run


Compilation

  • WRF compilation with dmpar
  • WRF compilation with dm+sm
  • WPS compilation with distributed memory with dm+sm compilation of WRF

Tests simulations


Tests simulations have been done on Jenkins. The simulations were:
  • First tutorial case (Jan 00) with 1 nest
  • First tutorial case (Jan 00) with 2 nests
  • First tutorial case (Jan 00) with additional diagnostics turned on.
WRF outputs for these tests can be found under /projects/WRF/data/KGO/. The inputs for the tests can be found in https://github.com/coecms/wrf-testing