Recent Changes

Monday, January 21

  1. page ERA5 edited ... This problem was caused by the computation of U/V from Vorticity (Vo) and Divergence (D) alrea…
    ...
    This problem was caused by the computation of U/V from Vorticity (Vo) and Divergence (D) already interpolated onto a regular grid in CDS. U/V should have been computed from Vo and D in the spherical harmonics representation (the native model representation) and then interpolated to a regular grid before saving to the climate data store.
    This error has yet to be fixed, which means that any data corresponding to the description above is wrong, and will continue to be wrong until a fix is in place. You should not use this data and wait until the problem is fixed.
    ItThe problem is estimatednow fixed and we are in the process of re-downloading the data.
    From the CDS communication:
    "During our investigations, we found
    that wind values are far too low on pressure levels at the problem will be fixedpoles in December 2018.CDS, which has been documented as a known issue. Please use winds at neighbouring locations at 89.75 (N/S) and 89.5 (N/S) for Reanalysis and Ensemble members winds instead of 90 (N/S)."
    License
    ERA5 data is open access and free to download for all uses, including commercial use.
    (view changes)
    12:28 pm

Wednesday, January 16

  1. file aug29_list.txt.zip (deleted) uploaded Deleted File
    9:52 pm
  2. file jun22_list.txt (deleted) uploaded Deleted File
    9:52 pm
  3. file jun15_list.txt (deleted) uploaded Deleted File
    9:52 pm
  4. file list.txt (deleted) uploaded Deleted File
    9:52 pm
  5. file jun06_list.txt (deleted) uploaded Deleted File
    9:52 pm
  6. file jun01_list.txt (deleted) uploaded Deleted File
    9:51 pm
  7. page Data FAQ edited ... The CMIP5 dataset is overall a Pb sized dataset contributed by as many as 60 different modelli…
    ...
    The CMIP5 dataset is overall a Pb sized dataset contributed by as many as 60 different modelling groups, hence it is inherently complicated to organise. As well because the modelling groups didn’t stick to the rules and did their own thing. Unfortunately, the guidelines in regard to versioning the dataset were not sufficiently detailed and so they've been interpreted differently by different groups. When the climate community started downloading data on Raijin it was decided that the only way to keep track of the dataset "version" was to re-create their DRS as in the web server (thredds) which are unique. We also always download the originally published dataset and no replicas from other nodes.
    When a web server reaches its capacity then the new datasets are published from a new server and this means that you could have a different root for the same model, sometimes for the same experiment.
    The CWSlab workflowCurrently NCI is a good way to find wherere-downloading the latest versions of CMIP5 non-Australian data is since it sorts outinto a more coherent directory structure. the new replicated data for you, same is for the scriptsstored in the CMS github repo CMIP5-utils. A list of all the files is updated weekly on Raijin: /g/data1/ua6/unofficial-ESG-replica/tmp/tree/esg-tree-LATEST-paths.txt. We are always working on wayal33 group. Refer to improve this, so feel free to provide feedback.their climate community page for information and updates.
    Do CMIP5 variables coming from the same simulations have the same version number?
    This question is difficult to answer and it is a really sore point with CMIP5, hopefully, they are implementing changes so it won't happen again in CMIP6.
    (view changes)
    9:44 pm
  8. page NCI Guidelines gdata edited Gdata is a disk storage with high-speed access. It is meant to be used to store data files used in …
    Gdata is a disk storage with high-speed access. It is meant to be used to store data files used in analysis. This disk is not backed up
    The location is always /g/dataN with the N identifying physically different disks, all of the Centre’s projects are on /g/data1 or on /g/data3. This means that they are effectively separate filesystems so if you’re moving files from a project in /g/data1 to a project in /g/data3 for example, you are effectively copying the files to a new filesystem. If you are moving files between two projects on the same disk, the transfer will be basically immediate because their physical location won’t change.
    Proper usage
    Gdata is used for large collections that users access often, so it is appropriate for:
    * PublishedPublished or shared
    ...
    (Reanalysis products), ua6 (CMIP5),rr3, al33 and oi10 (CMIP), ua8 (CoE
    ...
    fast access.
    * Model

    Model
    output or
    ...
    that analysis.
    * Software

    Software
    not otherwise
    ...
    on gdata:
    * Your

    Your
    own code
    ...
    original repository
    * Gdata

    Gdata
    is not
    *Any files that are no longer used after you finished to analyse your data you should “clean” it and archive it. If you think you might reuse the data in the future you should evaluate how likely this is and act accordingly. It’s very easy to leave the data there for the moment and never get back to it. Storage is becoming more and more scarce and this is by far the most expensive storage option.
    Good practices:
    * All

    All
    netcdf files
    ...
    be compressed
    * Use

    Use
    group permissions
    ...
    you can
    * Organise

    Organise
    your work
    ...
    plan, etc).
    * Clean

    Clean
    up regularly
    Big files:
    * A

    A
    big file
    ...
    several GB.
    * Avoid

    Avoid
    creating any
    ...
    techniques like:
    * Chunking

    Chunking
    (netcdf4): http://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters
    * Striping:

    Striping:
    https://opus.nci.org.au/display/Help/Lustre+Basics
    [[include component="page" wikiName="climate-cms" page="nci_guidelines_index" editable="1"]]
    (view changes)
    9:38 pm
  9. page Data edited ... - CMIP5 update: most of the ESGF nodes are back online, still missing are NCI and the Chinese …
    ...
    - CMIP5 update: most of the ESGF nodes are back online, still missing are NCI and the Chinese nodes.
    - CMIP5: the ARCCSSive module has now a new stable version and the database has been updated to include all the data replicated on raijin, both by users and by NCI as a bulk transfer.
    ...
    progress. Anyone belonging to ua6with an NCI login can access and also edit or add comments to some of the pages.
    Data policies and services within the Centre
    What data should I publish - Centre position statement:Preparing a Data Management Plan
    (view changes)
    9:32 pm

More