Publishing ACCESS model simulations to ESGF

There are two main phases to prepare an ACCESS simulation for publication on an ESG node: the post-processing and the quality control (QC).
Post-processing
In this phase we extract or calculate, from the raw model output, selected variables as defined by the CMIP5 standard_output table .
To do so we use the post-processor app created by Peter Uhe, CAWCR, this is a python wrapper to the Climate Model Output Rewriter (CMOR). CMOR2 is the current version and can be used to satisfy the specifications for CMIP5 model output.

APP1-0 description

APP1-0 is roughly made up of
  • database/champions – contains all the champions tables the files defining which variables will be calculated, their publishing status, the input variables, input filenames and, if applicable, how they will be calculated. There is a file for each CMIP table, plus a few more for “extra” variables. Fields are: cmip variable, definable in access, implemented in post processor, dimensions, access variable name, access file location, realm, calculation, override units, axes modifier, positive, notes. See at bottom of page for an example.
  • database/experiments.csv – this is the default name for a csv file in which the following attributes/settings are set for each simulation: experiment_id, experiment_directory, start_year, end_year, realization, initialization, physics_version, local_experiment name, model name, forcing, parent_id, parent_rip, branch time, land model, notes
  • database/app.db – this is the default name of a sqlite database that contains a combination of the information provided by the experiments.csv file and the champions tables. You can use the same app.db and experiments.csv file for all your simulations, we normally produced one for each experiment adding _local-exp-name to distinguish them. There is a row for each file that needs to be processed. The “status” field is updated after each processing step: unprocessed, processing, processed, published. And in case of errors: processing_failed.
  • database/database_manager.py is the python script that produces the app database. Which champions tables are used for different CMIP5 experiments (like historical, amip etc.) are defined here. If a database with the same name exists already it will update it
  • app_wrapper.py – this is a wrapper to app.py. It calls app.py and passes information about a specific simulation. It reads input information from the run app database (app.db default name) and set the version (ex. V20130607).
  • app.py – the main post-processor scripts, it calls CMOR passing to it all the relevant settings. In app.py some of the global attributes (such as title) are directly defined. App.py can call app_functs.py if a calculation needs to be performed to output a variable.
  • move_to_published.py – this script has two functions update the version and move the files to the “published” directory, currently /g/data1/p66/CMIP5/published/
This will become /g/data1/ua6/authoritative/IPCC or ACCESS_pre-publishing, at the moment the NCI publisher rsync the p66 dir to this one before starting the actual publishing. The “published” directory follows the official CMIP5 DRS. An example:
../CMIP5/output1/CSIRO-BOM/ACCESS1-3/historicalNat/mon/atmos/Amon/r1i1p1/files/cl_20130906/cl_Amon_ACCESS1-3_historicalNat_r1i1p1_185001-187412.nc
  • call.sh, run.sh, setup_env.sh – these are the scripts used to actually submit to the queue an instance of the post-processor and set up the environment to run it
  • /um2netcdf/ - this directory contains the script necessary to convert the UM binary output to netcdf

Set up of the APP1-0

  1. svn checkout https://trac.nci.org.au/svn/access_tools/post_processor/branches/APP1-0
  2. Update the run.sh, call.sh and setup_env.sh files so you are loading the correct environment variables and NCI project id. See at bottom of page for example scripts.
  3. Check that the settings in the champions tables are correct for your simulations. In particular: definable in access should be “yes” if you want to process the variable, implemented in post processor is a string representing the publishing priority and file location completes the input file path in the experiments.csv.
  4. In APP1-0/database there is an example of an experiments.csv file. Update this file with the information relating your simulations, you can use one for all simulation, adding a new row for each simulation, or, as we did, use separate ones for each of them. In particular this is where you set up the first part of the input files path.
  5. Set up the directory where you are storing the input files. Ex.
    • ../../<local-exp>/history/atm/netCDF/<UM output files>
    • ../../<local-exp>/history/ocn/<ocean output files>
    • ../../<local-exp>/history/ice/<sea-ice output files>
  6. Set up the directory where you are storing the output files and the ancillary files. Ex.
    • ../../<local-exp>/CMIP5/job_output/
    • ../../<local-exp>/CMIP5/ancillary_files/ {cice_grid_....nc, grid_spec.auscom_….nc, landFrac.nc, lsmask_.....nc, Base-09Ipj.astart-….nc}

Running the APP1-0
  1. source setup_env.sh – to load all the necessary modules and setting up the environment variables
  2. python database_manager.py – to create the app database
  3. ./run.sh – will submit to the queue call.sh which load the environment and run the app_wrapper.py python script.
  4. Check the run output and error log files. If there are errors or the process was interrupted and you need to reprocess a variable use the reset_db_status.py script in /database/ to reset the variable status to “unprocessed”. In case of error the status will be “processing_failed”, in case the job is interrupted before it can update the status, it will be “processing”.

Quality Control

The software to manage the QC component has been developed by DKRZ ( http://svn-mad.zmaw.de/svn/mad/Model/QualCheck). A copy of this is installed on raijin in /g/data1/p66/QC . The software consists of two components:
  • a set of c++ executables and scripts (QC) to do the actual checks (min,mean,max, etc)
  • a wrapper script (QCWrapper) to upload the results to a central postgressDb at DKRZ.

Move to publishing directory
  1. check settings for “outpath” if you’re not using the path set in the database. In our case we commented all the rows where the outpath was read from the app database, because the files have already being moved once to the standard ACCESS output directory. So we have explicitly defined outpath='/projects/p66/pfu599/'
  2. modify the select statement (around line 200 in code) so it select the fields you want to publish (use the priority field Ex. publish1 highest priority, etc. we have also added the publishing date to this field)
  3. run first selecting the update_version option (around line 210) if you need to.
  4. then run selecting the move_to_published option (around line 211), do this the first time as a dry_run, to check everything is ok, and then publish_var. You set these at lines 160-165 tmp=dry_run and tmp=publish_var

Quality Control set up
  1. go to /projects/p66/QC/QCWrapper and create your own running directory
  2. In this subdirectory copy the config file: qc_ACCESS_final.conf, and change
    • contact details and QCBD logfile at the start of the script (lines 18-24)
    • $QCBD_VERSION experiment version/s to process (around lines 30-50)
    • $DATA_ROOT_FS for the experiment, which has to end with the “experiment” subdir. Ex. DATA_ROOT_FS=/projects/p66/CMIP5/published/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historical/ (around lines 50-60)
  3. copy qc_call.sh in your own dir , change reference to configuration file (NB --nodb is the “no database” flag so it doesn't submit files to external database, --noqc is the “no quality control” flag). An example is shown at bottom of page.
  4. submit job to queue by running qsub qc_call.sh

After QC is finished
  1. Check log file /projects/p66/QC/QCWrapper/<your-subdir>/qcdb.log
  2. Check results under /projects/p66/QC/QCResults.ACCESS/CMIP5/output1/CSIRO-BOM/<<model>>/<<experiment>>/data/<<freq>>/<<realm>>/<<cmip-table>>/<<ensemble>>/<<version>>/<<var>>/
    • qc_<var>_<cmip-table>_<model>_<experiment>_<ensemble>.nc contains stats for the file
    • qc_warning_<var>_<cmip5-table>_<model>_<experiment>_<ensemble>_<period>.txt contains warning messages. You can safely ignore all the warnings about ACCESS1-3/ACCESS1-0 name.
    • tid_<var>_<cmip-table>_<model>_<experiment>_<ensemble>.txt contains the file tracking id and the error code (if 0 no errors)
  3. You can plot the file stats by copying in your area /g/data1/p66/QC/QCWrapper/pxp581/QC_scripts/qc_plot.py, this will produce .png files and store them in …/QC_scripts/qc_plots . On raijin you can use display <filename>.png to plot them. You can choose a database_plot option and you set the version of the experiment you want to process or a manual option that allows you to add more constraints to your selection.


Example of APP1-0/run.sh

#!/bin/bash -l
date=`date +%F_%H-%M-%S`
cd $HOME/APP1-0
#set up environment
. ./setup_env.sh
#Inputs
Exp=testHG3-HT # local experiment name
N=200 # maximum number of files that will be processed
Table=Limon # CMIP table to be processed
#Output files
name=a${Table}_${Exp}
dir=${APP_OUTPATH}/CMIP5/job_output/${Exp}/${Table}/
output=${dir}${date}_output.txt
error=${dir}${date}_error.txt
mkdir -p $dir
#qsub command
qsub -P w97 -q normal -l walltime=8:00:00,mem=16GB -lother=gdata1 -N $name -o $output -e $error -v Table=$Table,Exp=$Exp,N=$N ./call.sh
#command to use without environment variables
#qsub -P p66 -q normal -l walltime=8:00:00,mem=3500MB -wd -v ./call.sh

Example of APP1-0/call.sh

#!/bin/bash -l
cd $HOME/APP1-0
#set up environment
. ./setup_env.sh
#Call python script
python app_wrapper.py

Example of APP1-0/setup_env.sh

#!/bin/bash -l
#Set up environment for post-processor
  1. Global environment
module use ~access/modules
module load netcdf/4.3.0
module load python/2.7.3
module load python/2.7.3-matplotlib
module load pythonlib/cdat-lite/6.0rc2-fixed
module load pythonlib/ScientificPython/2.8
module load pythonlib/cmor
module load udunits
export UDUNITS2_XML_PATH=/apps/udunits/2.1.24/share/udunits/udunits2.xml
export PYTHONPATH=$PYTHONPATH:~/pythonlibs/lib/python2.7/site-packages/
  1. APP environment variables
export APP_OUTPATH=/g/data1/ua8/tmp/testHG3-HT/ #set this to where you want output data and logs to go
export APP_CHAMPIONS_DIR=$HOME/APP1-0/database/champions/
export APP_EXPERIMENTS_TABLE=experiments.csv
export APP_DATABASE=$HOME/APP1-0/database/app.db
export CDAT_LOCATION=~access/apps/pythonlib/cdat-lite/i6.0rc2-fixed/lib/python2.7/site-packages/cdat_lite-6.0rc2-py2.7-linux-x86_64.egg

Example of a champions table: day_limited.csv

#Champions table: day_limited,,,,,,,,,,,
#cmip variable,definable in access,implemented in post processor,dimensions,access variable name,access file location,realm,calculation,override units,dummy file,positive,notes
rhs,yes,pub2013-11,2Datmos,hurs,/atm/netCDF/*_dai.nc,atmos,,,,,
rhsmin,yes,pub2013-11,2Datmos,hursmin,/atm/netCDF/*_dai.nc,atmos,,,,,
rhsmax,yes,pub2013-11,2Datmos,hursmax,/atm/netCDF/*_dai.nc,atmos,,,,,
snc,no,unknown,2Datmos,,,landIce,,,,,
clt,yes,pub2013-11,2Datmos,cldlw,/atm/netCDF/*_dai.nc,atmos,,1,,,
tslsi,yes,failed,2Datmos,ts ts_sea
snw,yes,unknown,2Datmos,snw_land,/atm/netCDF/*_dai.nc,landIce,var[0].filled(0),,,,
prc,yes,pub2013-11,2Datmos,prc1 prc2,/atm/netCDF/*_dai.nc,atmos,var[0]+var[1],,,,
prsn,yes,pub2013-11,2Datmos,prsn,/atm/netCDF/*_dai.nc,atmos,,,,,
mrro,yes,need_check,2Datmos,mrros smrros,/atm/netCDF/*_dai.nc,land,var[0]+var[1],,,,"STASHmapping: m01s08i234 + m01s08i235, bad units: kg m-2 a-2"
uas,yes,pub2013-11,2Datmos,uas,/atm/netCDF/*_dai.nc,atmos,,,,,
vas,yes,pub2013-11,2Datmos,vas,/atm/netCDF/*_dai.nc,atmos,,,,,
sfcWindmax,yes,pub2013-11,2Datmos,wndmax,/atm/netCDF/*_dai.nc,atmos,,,,,
hfls,yes,pub2013-11,2Datmos,hfls,/atm/netCDF/*_dai.nc,atmos,,,,,
hfss,yes,pub2013-11,2Datmos,hfss,/atm/netCDF/*_dai.nc,atmos,,,,,
rlds,yes,pub2013-11,2Datmos,rlds,/atm/netCDF/*_dai.nc,atmos,,,,,
rlus,yes,pub2013-11,2Datmos,rls rlds,/atm/netCDF/*_dai.nc,atmos,(var[0]-var[1])*-1,,,,(m01s02i201-m01s02i207)*-1
rsds,yes,pub2013-11,2Datmos,rsds,/atm/netCDF/*_dai.nc,atmos,,,,,
rsus,yes,pub2013-11,2Datmos,rss rsds,/atm/netCDF/*_dai.nc,atmos,(var[0]-var[1])*-1,,,,(m01s01i201-m01s01i235)*-1
rlut,yes,pub2013-11,2Datmos,rlut_bl,/atm/netCDF/*_dai.nc,atmos,,,,,
usi,yes,pub2July,2Docean,uvel_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5
vsi,yes,pub2July,2Docean,vvel_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5
sic,yes,pub2July,2Docean,aice_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5
sit,yes,pub2July,2Docean,hi_d,/ice/iceh_day.????-??.nc,seaIce,,,,,Implement new namelist following ~pju565/ACCESS/input/cice/cice4.1_in.nml_0layer_cmip5
ta,yes,pub24_01_13,3Dp8,ta,/atm/netCDF/*_dai.nc,atmos,,,,,
hur,yes,pub24_01_13,3Dp8,hur
hus,yes,pub24_01_13,3Dp8,hus,/atm/netCDF/*_dai.nc,atmos,,,,,
wap,yes,pub24_01_13,3Dp8,wap,/atm/netCDF/*_dai.nc,atmos,,,,,
va,yes,pub24_01_13,3Dp8,va,/atm/netCDF/*_dai.nc,atmos,,,,,
ua,yes,pub24_01_13,3Dp8,ua,/atm/netCDF/*_dai.nc,atmos,,,,,
zg,yes,pub24_01_13,3Dp8,zg,/atm/netCDF/*_dai.nc,atmos,,,,,


Example of qc_call.sh to run QC check

#!/bin/bash
#PBS -l walltime=4:00:00
#PBS -l mem=1900mb
#PBS -j oe
#PBS -l wd
#PBS -o logs
module load python/2.7.3
module load python/2.7.3-matplotlib
module use ~access/modules
export PYTHONPATH=$PYTHONPATH:~pfu599/pythonlibs
export QCDB_TYPE=ORACLE
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/g/data1/p66/QC/Oracle/instantclient_11_1
python ./src/qcWrapper.py --configure=qc_ACCESS.conf --nodb