= The HPC quota system Harry Mangalam v1.00, Mar 5th, 2019 :icons: // fileroot="/home/hjm/nacs/hpcquotas"; asciidoc -a icons -a toc2 -a toclevels=3 -b html5 -a numbered ${fileroot}.txt; scp ${fileroot}.html ${fileroot}.txt moo:~/public_html == The overlapping problems We are simultaneously dealing with moving a lot of users from one large filesystem (/dfs1) to another (/dfs3) at the same time as we're changing the numeric Group IDs (GIDs) to conform to UCI's internal ID system so that users can transparently use HPC storage as well as CRSP storage (eventually). Your alphabetic GID eg: 'hamfisted' will not change but your numeric one (changed from '3663' -> '2457542') probably will, and the quota system uses the numeric one. Many of you have run into quota problems when you got assigned a new numeric GID because the quota system started allocating your storage to your primary GID which caused an overflow: 'Quota Exceeded'. [[differentquotas]] .Different Quota systems [NOTE] ======================================================= There is a quota system for each different type of filesystem. - 'quota', which will list your quota on your HOME dir and on other NFSs (Network File Systems). - 'dfsquotas' which is a wrapper around a more complicated quota command for the BeeGFS filesystems (aka /dfs1 (now end-of-lifed), /dfs2, /dfs3, /sbak) ------------------------------------------------------- # use as: $ dfsquotas userid which-dfs # ie $ dfsquotas hmangala dfs3 ------------------------------------------------------- ======================================================= == Demonstrating with the 'khoih' user The 'id' command emits the entire membership of the user: the USER ID aka uid (1159269), as well as the primary GROUP ID aka gid (1159269). Note that they have the same number (1159269), but that's not always the case. It also shows all the other groups that khoih belongs to: 'groups= ...' ------------------------------------------------------- $ id khoih uid=1159269(khoih) gid=1159269(khoih) groups=123(bio),126(krt),473(class-ee282),544(longlab),2968(khoih_q),1025(tdlong_s), 1159269(khoih) ------------------------------------------------------- When we changed the GID (now 1159269(khoih)) and moved 'khoih' to /dfs3, everyone got a 2TB quota which was set on your GID. That's the problem. To see the problematic result, scroll down to the link:#overquota[error comment] below The following is the full output of the 'dfsquotas' command, showing *all* the quotas associated with the user, USER first, then all the GROUPs. ------------------------------------------------------------------------------ $ dfsquotas khoih dfs3 # you can execute this command safely ===== [USER quota: khoih on dfs3] Quota information for storage pool Default (ID: 1): user/group || size || chunk files name | id || used | hard || used | hard -----------|------||------------|------------||---------|--------- khoih|1159269|| 7.46 TiB| unlimited|| 696208|unlimited ==== [Group Quotas on dfs3] ==== Group quota [bio] Quota information for storage pool Default (ID: 1): user/group || size || chunk files name | id || used | hard || used | hard ------------|------||------------|------------||---------|--------- bio| 123|| 93.91 TiB| 126.00 TiB|| 8794907|unlimited ==== Group quota [krt] Quota information for storage pool Default (ID: 1): user/group || size || chunk files name | id || used | hard || used | hard ------------|------||------------|------------||---------|--------- krt| 126|| 561.85 GiB| 50.00 TiB|| 501949|unlimited ------------------------------------------------------------------------------ *.. Output deleted..* ------------------------------------------------------------------------------ ==== Group quota [tdlong_s] Quota information for storage pool Default (ID: 1): user/group || size || chunk files name | id || used | hard || used | hard ------------|------||------------|------------||---------|--------- tdlong_s| 1025|| 13.16 TiB| 50.00 TiB|| 693964|unlimited ------------------------------------------------------------------ [[overquota]] *here's the problem: the new GID is exceeding your default 2TB quota* ------------------------------------------------------------------ ==== Group quota [khoih] Quota information for storage pool Default (ID: 1): user/group || size || chunk files name | id || used | hard || used | hard ------------|------||------------|------------||---------|--------- khoih|1159269|| 7.46 TiB| 2.00 TiB|| 696207|unlimited ^^^^^^^^ > ^^^^^^^^ ------------------------------------------------------------------ == How to fix All is not lost: You can tell HPC to store some of these files as another group if you are members of both groups. === chgrp ie if you want to change the ownership of some of the files to 'som' an example in my dir tree: [source,bash] ------------------------------------------------------------------ # list the ownership of file 'usr_lib_cgi-bin.tgz' $ ls -l usr_lib_cgi-bin.tgz -rw-r--r-- 1 hmangala staff 30265 Jul 6 2010 usr_lib_cgi-bin.tgz # +++++ -- group ownership is 'staff' # now change the group to 'som' $ chgrp som usr_lib_cgi-bin.tgz # list again $ ll usr_lib_cgi-bin.tgz -rw-r--r-- 1 hmangala som 30265 Jul 6 2010 usr_lib_cgi-bin.tgz # +++ -- group ownership is now 'som' ------------------------------------------------------------------ Importantly, the quota will now show up in the 'som' group quota. === newgrp Another command is 'newgrp'. See 'man newgrp' This command changes the group under which you are logged in and therefore changes the ownership of the files which you create when logged in. For example you could use the 'newgrp' command in your qsub file to make sure that all the files created during a batch job are owned by the correct group and not by your primary group. ie: [source,bash] ------------------------------------------------------------------ # demonstrating as 'hmangala' # here's my id membership $ id hmangala uid=785(hmangala) gid=200(staff) groups=200(staff),134(stata),140(som),401(macvaw),418(clc) # and if I 'touch' a file into being $ touch file1 # it's has my original group ID ownership $ ll file1 -rw-r--r-- 1 hmangala staff 5290 Mar 6 16:57 file1 # +++++ # now use 'newgrp' to log in under the 'macvaw' group $ newgrp macvaw $ id hmangala # my id membership doesn't change uid=785(hmangala) gid=200(staff) groups=200(staff),134(stata),140(som),401(macvaw),418(clc) # but when I create new files... $ touch file2 # the new file has the macvaw group ownership $ ls -l file2 -rw-r--r-- 1 hmangala macvaw 0 Mar 6 15:54 file2 ++++++ ------------------------------------------------------------------