It is important for everyone to regularly check their usage (storage and SU). Since most file systems are shared (except /home), exceeding a quota means the work for everyone on the project is stalled. Monitoring your usage and finding the limits is different for each filesystem. We'll quickly present the different ways here as well as ways to monitor your SU usage.

nci_account


This command shows you a snapshot of:
  • SU usage and a breakdown by queue.
  • /short, /g/data and massdata (or mdss) usage
This command is maintained by NCI and is available by default on Raijin.

Example:

$ nci_account -P w35
 
Usage Report: Project=w35 Compute Period=2017.q4 (01/10/2017-31/12/2017)
========================================================================
 
Total Grant: 725.00 KSU
Total Used:  216.90 KSU
Total Avail: 508.10 KSU
Bonus Used:   0.00 KSU
 
---------------------------------------------------------------------------------------------------------------
System        Queue  Charge         Usage         Usage       SU used   Reserved SU    Pending SU      Total SU
                     Weight     (CPU Hrs)    (Walltime)                   (Running)      (Queued)     Committed
raijin          knl    0.25          0.00          0.00          0.00          0.00          0.00          0.00
raijin      hugemem    1.25          0.00          0.00          0.00          0.00          0.00          0.00
raijin      megamem    1.25          0.00          0.00          0.00          0.00          0.00          0.00
raijin       normal    1.00     170468.70     185010.50     185010.50       3840.00          0.00     188850.50
raijin     normalsp    1.25          0.00          0.00          0.00          0.00          0.00          0.00
raijin        copyq    1.00         31.01         46.72         46.72         20.00         10.00         76.72
raijin      express    3.00       4394.84       5281.46      15844.39          0.00          0.00      15844.39
raijin    expressbw    3.75          5.06          8.90         33.37          0.00          0.00         33.37
raijin    gpupascal    4.00          0.00          0.00          0.00          0.00          0.00          0.00
raijin     fusionio    1.00          0.00          0.00          0.00          0.00          0.00          0.00
raijin          gpu    3.00          0.00          0.00          0.00          0.00          0.00          0.00
raijin     normalbw    1.25      12705.24      12770.99      15963.73          0.00          0.00      15963.73
---------------------------------------------------------------------------------------------------------------
Overall                         187604.86     203118.56     216898.70       3860.00         10.00     220768.70
 
 
Usage Report: Project=w35 Storage Period=2017.11 (01/10/2017-31/12/2017)
========================================================================
-------------------------------------------------------------------------------------------------
System    StoragePt             Grant       Usage       Avail      iGrant      iUsage      iAvail
-------------------------------------------------------------------------------------------------
dmf       massdata          4048.00GB   1209.00GB   2839.00GB     323.00K       8.88K     314.12K
global    gdata1a             76.00TB     63.28TB     12.72TB    3883.00K    2356.15K    1526.85K
raijin    short               15.00TB     11.58TB      3.42TB    3280.00K    2692.70K     587.30K
-------------------------------------------------------------------------------------------------
Total                         94.95TB     76.04TB     18.91TB    7486.00K    5057.73K    2428.27K
 

Useful options:

  • -v shows a breakdown per allocation source of SU and storage usage and per user of SU usage.
  • -p allows you to select a period:
    • YYYY.qX to select a specific quarter
    • YYYY.X to select a specific month
    • YYYY for the total over a year (or up to today for current year).

short_files_report


This command gives you a snapshot of:
  • the storage per user owned by a project
OR
  • the storage per user under /short/<project>
This command is maintained by NCI and is available by default on Raijin.
Useful options
  • -G gives you the breakdown per user and location of all files owned by a project. This is usually the useful option when looking at quota issues.
Example:
$ short_files_report -G w35
Usage details for project w35:
---------------------------------------------------
          folder   user   size    inodes  scan date
---------------------------------------------------
             z00 hxw599 5996kB        32 2017-11-29
             k10 kxl561 28.0kB         7 2017-11-29
             n81 gdv561 2170MB     17568 2017-11-29
             p66 saw562 84.0kB        19 2017-11-29
             p66 kxl561 28.0kB         7 2017-11-29
             q90 saw562 55.4MB        28 2017-11-29
             r99 dxa561  478GB      3739 2017-11-29
             r99 mxs561 11.5GB      8947 2017-11-29
#...
---------------------------------------------------
                 totals 12.0TB   2794317
---------------------------------------------------
Due to the overheads involved in trawling the
entire filesystem, this information should only be
used as a guide, and may be out of date.
---------------------------------------------------
 
The first column indicates in which project area the files are located. Hence, the first line means user hxw599 owns 32 files for a total size of 5996kb owned by w35 but stored under /short/z00.

  • -P gives you the breakdown per user and group ownership of all files located in a /short/<project> area.
Example:
$ short_files_report -P w35
Usage details for /short/w35:
---------------------------------------------------
           group   user   size    inodes  scan date
---------------------------------------------------
             x77 saw562 82.4MB        27 2017-11-29
             k10 saw562 6929MB      6183 2017-11-29
             k10 jtt565  416GB      1758 2017-11-29
          access saw562 2458MB    104144 2017-11-29
          access mrd561 1430MB     96071 2017-11-29
             n81 mxs561  660MB      2048 2017-11-29
             p66 saw562 2618MB      1226 2017-11-29
             p66 hxw599  439MB        57 2017-11-29
             q90 saw562 7892kB        14 2017-11-29
#...
---------------------------------------------------
                 totals 12.7TB   3014008
---------------------------------------------------
Due to the overheads involved in trawling the
entire filesystem, this information should only be
used as a guide, and may be out of date.
---------------------------------------------------
 
The first column indicates which group owns the data. Hence the first line means user saw562 owns 27 files for a total of 82.4MB in /short/w35 but owned by project x77

gdataX_files_report


These commands give you a snapshot of:
  • the storage per user owned by a project
OR
  • the storage per user under /g/dataX/<project>
There are 3 such commands: gdata1a_files_report, gdata2_files_report, gdata3_files_report. These commands are maintained by NCI and are available by default on Raijin.
Useful options
  • -G gives you the breakdown per user and location of all files owned by a project. This is usually the useful option when looking at quota issues.
Example:
$ gdata1a_files_report -G w35
Usage details for project w35:
---------------------------------------------------
          folder   user   size    inodes  scan date
---------------------------------------------------
          access saw562  416MB     29768 2017-11-29
          access hxw599 14.7MB       173 2017-11-29
             p66 pxp581 68.0kB        10 2017-11-29
             w35 aph502 3810MB        11 2017-11-29
             w35 pxp581  511MB    110855 2017-11-29
             w35 saw562 4002GB     22741 2017-11-29
             w35 hxw599 69.4GB        65 2017-11-29
             w35 ccc561  865GB      5968 2017-11-29
             w35 mrd561 23.4TB   1722930 2017-11-29
             w35 mgk576  4.0kB         1 2017-11-29
             w35 mxs561 32.1TB    411506 2017-11-29
             w35 amu561 2981GB     37290 2017-11-29
             w35 ms8355 10.3GB        31 2017-11-29
             ua6 pxp581 60.0kB         4 2017-11-29
             ua8 pxp581  744kB         2 2017-11-29
      access.dev saw562 3004kB       421 2017-11-29
             wd9 nnh561 45.3GB     14347 2017-11-29
             ub4 pxp581 38.5MB        22 2017-11-29
---------------------------------------------------
                 totals 63.3TB   2356145
---------------------------------------------------
Due to the overheads involved in trawling the
entire filesystem, this information should only be
used as a guide, and may be out of date.
---------------------------------------------------
The first column indicates in which project area the files are located. Hence, the first line means user saw562 owns 29768 files for a total size of 416MB owned by w35 but stored under /g/data1a/access

  • -P gives you the breakdown per user and group ownership of all files located in a /g/data/<project> area.
Example:
$ gdata1a_files_report -P w35
Usage details for /g/data1a/w35:
---------------------------------------------------
           group   user   size    inodes  scan date
---------------------------------------------------
            UNSW mxs561  8.0kB         2 2017-11-29
             k10 hxw599 4689MB         4 2017-11-29
          access mxs561  207MB     15435 2017-11-29
             hh5 mxs561  140GB      1465 2017-11-29
             w35 aph502 3810MB        11 2017-11-29
             w35 pxp581  511MB    110855 2017-11-29
             w35 saw562 4002GB     22741 2017-11-29
             w35 hxw599 69.4GB        65 2017-11-29
             w35 ccc561  865GB      5968 2017-11-29
             w35 mrd561 23.4TB   1722930 2017-11-29
             w35 mgk576  4.0kB         1 2017-11-29
             w35 mxs561 32.1TB    411506 2017-11-29
             w35 amu561 2981GB     37290 2017-11-29
             w35 ms8355 10.3GB        31 2017-11-29
             x45 vxh599 25.4MB      2585 2017-11-29
             dt6 amu561  185GB    118137 2017-11-29
---------------------------------------------------
                 totals 63.6TB   2449026
---------------------------------------------------
Due to the overheads involved in trawling the
entire filesystem, this information should only be
used as a guide, and may be out of date.
---------------------------------------------------
The first column indicates which group owns the data. Hence the first line means user mxs561 owns 2 files for a total of 8.0kB in /g/data1a/w35 but owned by project UNSW

ncimonitor


This command gives you plots of:
  • SU usage over time
  • /short usage over time breakdown per user
  • /g/data usage over time breakdown per user
  • SU usage over time breakdown per user
  • change in /short and /g/data usage over time per user
This command is maintained by CMS. To use:
module purge
module use /g/data3/hh5/public/modules
module load conda/analysis27
You can access help on https://github.com/coecms/ncimonitor or via command line:
ncimonitor -h

ncfind and nccompress


These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them.
You'll find a complete documentation on this wiki page

du, find and wc


These commands can help you identify where in your area are the large files or which directories contain lots of files. These are Unix commands and all have man pages for information.
Useful options:
  • Check the sizes of sub-directories:
du -shc *
  • Find files larger than X
find . -type f -size +100M
  • Count number of files and subdirectories in directory
ls -1 | wc -l
Note: the Internet is the perfect source for commands and options variations to find this type of information on your files and directories.

ncdu


ncdu is a disk usage analyzer with an ncurses interface. It is designed to find space hogs and aims to be fast, simple and easy to use.

This command is part of the conda environment maintained by CMS. To use:
module purge
module use /g/data3/hh5/public/modules
module load conda/analysis3

To invoke run ncdu directory_path and replace directory_path with a path to a directory you wish to check.

Some example output:
ncdu /g/data3/hh5/public/apps
Gives a screen items in the directory sorted by size:
--- /g/data3/hh5/public/apps -----------------------------------------------------------------------------------------------------------
   64.0 GiB [##########] /miniconda3
  799.6 MiB [          ] /lrose
  152.3 MiB [          ] /easybuild
    4.0 KiB [          ]  README
 
 
 
 Total disk usage:  64.9 GiB  Apparent size:  64.6 GiB  Items: 2461484
Pushing <g> twice changes the way the usage is displayed
ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps -----------------------------------------------------------------------------------------------------------
   64.0 GiB [ 98.6% ##########] /miniconda3
  799.6 MiB [  1.2%           ] /lrose
  152.3 MiB [  0.2%           ] /easybuild
    4.0 KiB [  0.0%           ]  README
 
 
 Total disk usage:  64.9 GiB  Apparent size:  64.6 GiB  Items: 2461484
Select a directory, push <return> and it will show you a view of that directory sorted by size
ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /g/data3/hh5/public/apps/miniconda3 ------------------------------------------------------------------------------------------------
                                /..
   33.2 GiB [ 51.9% ##########] /pkgs
   22.6 GiB [ 35.3% ######    ] /envs
   11.8 GiB [ 18.5% ###       ] /old-envs
    4.6 GiB [  7.1% #         ] /conda-bld
  236.6 MiB [  0.4%           ] /lib
    8.7 MiB [  0.0%           ] /share
    6.1 MiB [  0.0%           ] /bin
    5.1 MiB [  0.0%           ] /include
    4.9 MiB [  0.0%           ] /conda-meta
    4.3 MiB [  0.0%           ] /ssl
    1.9 MiB [  0.0%           ] /compiler_compat
   28.0 KiB [  0.0%           ] /etc
   12.0 KiB [  0.0%           ] /x86_64-conda_cos6-linux-gnu
    4.0 KiB [  0.0%           ] /locks
    4.0 KiB [  0.0%           ]  LICENSE.txt
 
 
 
 Total disk usage:  64.0 GiB  Apparent size:  63.7 GiB  Items: 2446705
Select `/..` at the top to move back up the directory tree.

In this way it is possible to navigate the directory structure and pinpoint the largest users of disk space.

The disk usage information is cached, so once it has analysed a directory tree it is very fast to navigate.

Press <q> to quit the program.

NCI Guidelines Index

General: NCI Guidelines | Requesting Access | NCI website | NCI help| Connecting without password | Additional software on Raijin
Storage: /home | /short | /g/data | MDSS
Allocations: Projects | Compute Allocations | Monitoring Usage | Model performance