The HPC quota system

1. The overlapping problems

We are simultaneously dealing with moving a lot of users from one large filesystem (/dfs1) to another (/dfs3) at the same time as we’re changing the numeric Group IDs (GIDs) to conform to UCI’s internal ID system so that users can transparently use HPC storage as well as CRSP storage (eventually). Your alphabetic GID eg: hamfisted will not change but your numeric one (changed from 3663 → 2457542) probably will, and the quota system uses the numeric one.

Many of you have run into quota problems when you got assigned a new numeric GID because the quota system started allocating your storage to your primary GID which caused an overflow: Quota Exceeded.

Different Quota systems

There is a quota system for each different type of filesystem.

quota, which will list your quota on your HOME dir and on other NFSs (Network File Systems).
dfsquotas which is a wrapper around a more complicated quota command for the BeeGFS filesystems (aka /dfs1 (now end-of-lifed), /dfs2, /dfs3, /sbak)

# use as:
$ dfsquotas userid which-dfs

# ie
$ dfsquotas hmangala dfs3

2. Demonstrating with the khoih user

The id command emits the entire membership of the user: the USER ID aka uid (1159269), as well as the primary GROUP ID aka gid (1159269). Note that they have the same number (1159269), but that’s not always the case. It also shows all the other groups that khoih belongs to: groups= …

$ id khoih
uid=1159269(khoih) gid=1159269(khoih) groups=123(bio),126(krt),473(class-ee282),544(longlab),2968(khoih_q),1025(tdlong_s),
1159269(khoih)

When we changed the GID (now 1159269(khoih)) and moved khoih to /dfs3, everyone got a 2TB quota which was set on your GID. That’s the problem.

To see the problematic result, scroll down to the error comment below The following is the full output of the dfsquotas command, showing all the quotas associated with the user, USER first, then all the GROUPs.

$ dfsquotas khoih dfs3   # you can execute this command safely


===== [USER quota: khoih on dfs3]

Quota information for storage pool Default (ID: 1):

     user/group     ||           size          ||    chunk files
    name     |  id  ||    used    |    hard    ||  used   |  hard
  -----------|------||------------|------------||---------|---------
        khoih|1159269||    7.46 TiB|   unlimited||   696208|unlimited


==== [Group Quotas on dfs3]

==== Group quota [bio]

Quota information for storage pool Default (ID: 1):

     user/group     ||           size          ||    chunk files
    name     |  id  ||    used    |    hard    ||  used   |  hard
 ------------|------||------------|------------||---------|---------
          bio|   123||   93.91 TiB|  126.00 TiB||  8794907|unlimited

==== Group quota [krt]

Quota information for storage pool Default (ID: 1):

     user/group     ||           size          ||    chunk files
    name     |  id  ||    used    |    hard    ||  used   |  hard
 ------------|------||------------|------------||---------|---------
          krt|   126||  561.85 GiB|   50.00 TiB||   501949|unlimited

.. Output deleted..

==== Group quota [tdlong_s]

Quota information for storage pool Default (ID: 1):

     user/group     ||           size          ||    chunk files
    name     |  id  ||    used    |    hard    ||  used   |  hard
 ------------|------||------------|------------||---------|---------
     tdlong_s|  1025||   13.16 TiB|   50.00 TiB||   693964|unlimited

here’s the problem: the new GID is exceeding your default 2TB quota

==== Group quota [khoih]

Quota information for storage pool Default (ID: 1):

     user/group     ||           size          ||    chunk files
    name     |  id  ||    used    |    hard    ||  used   |  hard
 ------------|------||------------|------------||---------|---------
        khoih|1159269||    7.46 TiB|    2.00 TiB||   696207|unlimited
                           ^^^^^^^^  >  ^^^^^^^^

3. How to fix

All is not lost: You can tell HPC to store some of these files as another group if you are members of both groups.

3.1. chgrp

ie if you want to change the ownership of some of the files to som

an example in my dir tree:

# list the ownership of file 'usr_lib_cgi-bin.tgz'
$ ls -l usr_lib_cgi-bin.tgz
-rw-r--r-- 1 hmangala staff 30265 Jul  6  2010 usr_lib_cgi-bin.tgz
#                     +++++ -- group ownership is 'staff'

# now change the group to 'som'
$ chgrp som usr_lib_cgi-bin.tgz

# list again
$ ll usr_lib_cgi-bin.tgz
-rw-r--r-- 1 hmangala som 30265 Jul  6  2010 usr_lib_cgi-bin.tgz
#                     +++ -- group ownership is now 'som'

Importantly, the quota will now show up in the som group quota.

3.2. newgrp

Another command is newgrp. See man newgrp

This command changes the group under which you are logged in and therefore changes the ownership of the files which you create when logged in. For example you could use the newgrp command in your qsub file to make sure that all the files created during a batch job are owned by the correct group and not by your primary group.

ie:

# demonstrating as 'hmangala'
# here's my id membership
$ id hmangala
uid=785(hmangala) gid=200(staff) groups=200(staff),134(stata),140(som),401(macvaw),418(clc)

# and if I 'touch' a file into being
 $ touch file1

# it's has my original group ID ownership
$ ll file1
-rw-r--r-- 1 hmangala staff 5290 Mar  6 16:57 file1
#                     +++++


# now use 'newgrp' to log in under the 'macvaw' group
$ newgrp macvaw

$ id hmangala  # my id membership doesn't change
uid=785(hmangala) gid=200(staff) groups=200(staff),134(stata),140(som),401(macvaw),418(clc)

# but when I create new files...
$ touch file2

# the new file has the macvaw group ownership
$ ls -l file2
-rw-r--r-- 1 hmangala macvaw 0 Mar  6 15:54 file2
                      ++++++