Harry's HOWTOs ============== Harry Mangalam v1.27, Jan 30, 2019 :icons: // fileroot="/home/hjm/FF/HOWTOS"; asciidoc -a icons -a toc2 -a toclevels=3 -b html5 -a numbered ${fileroot}.txt; scp ${fileroot}.html ${fileroot}.txt moo:~/public_html //This should be organized into sections, like // Scientific Apps // R // System Administration // Disks and RAID // mdadm // hardware raid // Networking // Users // Programming Languages // Perl // Python // C // SQL // #################### System #################### // [[hpccluster]] == HPC hints & tweaks === callee script breakage There's a script that runs on new HPC logins to have users self-identify for Garr's accounting program, called: /data/hpc/bin/hpc-user-self-identifying-callee (authored in the mists of time by Adam Brenner, modified by Edward). which further calls a python routine that calls UCI's LDAP server to identify the user's PI: /data/hpc/accounting/user-identification/hpc-user-self-identifying.py Since the LDAP change-over, the LDAP call has been failing each time its run, confusing users and generating support tix. It's now fixed (until the next LDAP changeover). Also, this script is triggered if there's not a user-named lockfile in /data/hpc/accounting/user-identification/locks/ Somehow these locks have mostly become owed by 'root:staff' instead of by the user, which causes errors should they try to re-run the script. I haven't changed that since probably very few ppl will try that. just an FYI. === gnuplot file sizes --------------------------------------------------- module purge; module load perl/5.26.1; module load gnuplot cd # fllowing also tees the 'ls -lR' into a file of same name # to save the data should anything explode. ls -lR | scut -f=4 | tee ls-lR | sort -gr | feedgnuplot \ --extracmds 'set logscale y' --------------------------------------------------- === Restart KDE Window Manager -------------------------------------------------- kwin --restart & # simple! -------------------------------------------------- and the plasmashell (kpanel): -------------------------------------------------- kquitapp plasmashell && kstart plasmashell # or kbuildsycoca5 && kquitapp plasmashell && kstart plasmashell -------------------------------------------------- === Setting Core Dumps Because we don't want everyone to be dumping corefiles all over the place, we allow coredumps on a one-off basis. Change the user in the file '/data/shell-syswide-setup/system-wide-bashrc'. The section below shows user 'anassar' -------------------------------------------------- if [ "$USER" != "anassar" ]; then ulimit -c 0 fi -------------------------------------------------- === environment modules See http://modules.sourceforge.net/man/modulefile.html[this page] for all the module-related stuff. == System Administration === loadavg exceptions Sometimes loadavg goes berserk for various reasons. The most frequent offender leading to this state is not things using the CPU since that's obvious, but processes stuck in limbo, usually due to network disk mount problems. For example, we often have NFS disks that go walkabout and when users try to do an 'ls' or 'du' on those mounts, the process just hangs, driving up the loadavg, 1 unit per hung command. What commands are doing this is often difficult to discern, but the 'ps' command has an option that will help. From our CentOS 6.9 system ---------------------------------------------------------- PROCESS STATE CODES Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process. State Explanation D Uninterruptible sleep (usually IO) R Running or runnable (on run queue) S Interruptible sleep (waiting for an event to complete) T Stopped, either by a job control signal or because it is being traced. W paging (not valid since the 2.6.xx kernel) X dead (should never be seen) Z Defunct ("zombie") process, terminated but not reaped by its parent. ---------------------------------------------------------- On another system, with slightly different explanations ---------------------------------------------------------- State Explanation D Marks a process in disk (or other short term, uninterruptible) wait. I Marks a process that is idle (sleeping for longer than about 20 seconds). L Marks a process that is waiting to acquire a lock. R Marks a runnable process. S Marks a process that is sleeping for less than about 20 seconds. T Marks a stopped process. W Marks an idle interrupt thread. Z Marks a dead process (a "zombie"). ---------------------------------------------------------- The *D* status flag in a 'ps' output indicates that the process is 'disk wait' state - waiting for the disk to respond. If it's a hung NFS disk mount, NFS has the "feature" that it will not return a 'fugedaboutit, I'm dead', but a 'just stepped out, will be back soon, please wait' signal so the requesting process will do just that. In some cases (NFS server rebooting),this is a good thing. In others (NFS server power supplies dead), not so much. The extra apparent 'loadavg' doesn't affect the actual performance of the requesting server, but it screws up a lot of stats if you're trying to use 'loadavg' as an actual server load metric which many monitoring systems do (ganglia, for one). So to track down the offending processes, search for those processes which are waiting on disk. -------------------------------------------------------- ps -e u | grep ' D ' # find all processes waiting on Disk ps -e u | grep ' Ds ' # ditto -------------------------------------------------------- and kill them off. In many cases the 1st will find 'ls' and 'du'. The 2nd will often find instances of 'bash' which have hung wating for disk reads to complete. On our cluster, on a login node with 16 cores, which had a loadavg of 24, killing off those processes that were waiting on disk (using the 'D' identifier) dropped the loadavg from 24 to 14. When I killed off those procs with 'Ds', loadavg continued to drop to the normal range ~ 1.5. See https://goo.gl/uF8Hnf[here] === How to figure out Hardware It's often useful to figure out what's in a system remotely or without breaking open the case, or to catalog/query various bits of internal hardware. Herewith, 3 methods for doing so. - lspci - brief overview, doesn't need root, can dump some interesting info such as with '-k' the kernel driver for that hardware, '-vvv' for everything known about the device - lshw - more detailed, doesn't really correspond to physical layout. Also supports classes, but in a different way. Use 'lshw -short | less' to get a short view with Class of device. - dmidecode - super detailed, should use DMI types to list the sections, otherwise it's VERY long. (ie '-t 9' are the PCI slots); corresponds better to the physical layout of the system. === Sensors Scan all the sensors on a (modern) machine. Down to the NVME SSDs. -------------------------------------------------------- $ ipmitool sensor CPU1 Temp | 59.000 | degrees C | ok | 5.000 | 5.000 CPU2 Temp | 58.000 | degrees C | ok | 5.000 | 5.000 System Temp | 27.000 | degrees C | ok | 5.000 | 5.000 Peripheral Temp | 37.000 | degrees C | ok | 5.000 | 5.000 MB_NIC_Temp1 | na | | na | na | na MB_NIC_Temp2 | na | | na | na | na VRMCpu1 Temp | 37.000 | degrees C | ok | 5.000 | 5.000 VRMCpu2 Temp | 38.000 | degrees C | ok | 5.000 | 5.000 VRMSoc1 Temp | 52.000 | degrees C | ok | 5.000 | 5.000 VRMSoc2 Temp | 52.000 | degrees C | ok | 5.000 | 5.000 VRMP1ABCD Temp | 42.000 | degrees C | ok | 5.000 | 5.000 VRMP1EFGH Temp | 32.000 | degrees C | ok | 5.000 | 5.000 VRMP2ABCD Temp | 33.000 | degrees C | ok | 5.000 | 5.000 VRMP2EFGH Temp | 43.000 | degrees C | ok | 5.000 | 5.000 P1-DIMMA1 Temp | na | | na | na | na P1-DIMMA2 Temp | 39.000 | degrees C | ok | 5.000 | 5.000 P1-DIMMB1 Temp | na | | na | na | na ... -------------------------------------------------------- === OS Maint ==== Change Dir to own files This changes the ownership to sticky so that any files writ here will be owned by the dir owner. -------------------------------------------------------- chmod g+s dir -------------------------------------------------------- ==== Boot single, ignoring fstab From http://www.fclose.com/b/linux/2500/bypassing-bad-fstab-failure-while-booting-linux/[here]. Interrupt the boot (usually 'e') to edit the kernel boot line and add the following to the kernel line -------------------------------------------------- kernel .... single rw init=/bin/bash -------------------------------------------------- that should boot to single user mode, IGNORING the '/etc/fstab' enabling you to change it to correct a bad fstab entry (but haven't tried it yet..) ==== Get MacOSX OS ver # ------------------------------------------------------------ $ uname -a Darwin flop.nac.uci.edu 9.7.0 Darwin Kernel Version 9.7.0: Tue Mar 31 22:54:29 PDT 2009; root:xnu-1228.12.14~1/RELEASE_PPC Power Macintosh # or $ sw_vers ProductName: Mac OS X ProductVersion: 10.5.7 BuildVersion: 9J61 ------------------------------------------------------------ ==== Install MacOSX pkgs via cmdline if the disk image *OSXvnc1.71.dmg* is downloaded at /Users/hjm ----------------------------------------------------------------------------- 12:24:22 hjm@cg1 ~ 72 $ hdiutil attach OSXvnc1.71.dmg Checksumming Driver Descriptor Map (DDM : 0)... Driver Descriptor Map (DDM : 0): verified CRC32 $876DBC1A Checksumming Apple (Apple_partition_map : 1)... Apple (Apple_partition_map : 1): verified CRC32 $3FC18960 Checksumming disk image (Apple_HFS : 2)... disk image (Apple_HFS : 2): verified CRC32 $3A4E8BDD Checksumming (Apple_Free : 3)....... (Apple_Free : 3): verified CRC32 $00000000 verified CRC32 $3A457474 /dev/disk1 Apple_partition_scheme /dev/disk1s1 Apple_partition_map /dev/disk1s2 Apple_HFS /Volumes/OSXvnc 12:24:38 hjm@cg1 ~ 73 $ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/disk0s3 159955416 153355312 6344104 97% / devfs 1 1 0 100% /dev fdesc 1 1 0 100% /dev 512 512 0 100% /.vol /dev/disk1s2 10408 9528 880 92% /Volumes/OSXvnc ----------------------------------------------------------------------------- if it's the standard MacOSX app, it is actually a folder which has all the bits inside it and it can be installed simply by copying it to the /Applications folder. ==== Reconfigure exim Don't bother trying to hand-edit all the options in the config file. MUCH easier to just run the reconfig routine and type in the correct settings. After trying to reset the hostname for many minutes but not being able to find it, I just re-ran this and all was well. ---------------------------------------------------------------------------- dpkg-reconfigure exim4-config # (as root) ---------------------------------------------------------------------------- ==== Make an 'initrd' image to match your new kernel yaird is easiest: ----------------------------------------------------------------------------- yaird --verbose --output=/boot/initrd.img-2.6.22.1 2.6.22.1 ----------------------------------------------------------------------------- (but got a fatal error the last time). However the update-initramfs tool from the initramfs-tools package. ----------------------------------------------------------------------------- update-initramfs -k 2.6.22.1 -c -v ----------------------------------------------------------------------------- (spits out a lot of info about the build, but seems to work just fine). ==== Updating SL6.2 repos http://www.dedoimedo.com/computers/scientific-linux-repos.html[see here]. ==== Replicate a Debian system It's often convenient to be able to re-load all the packages from one syst to another (replicating an exisiting system, post-major upgrade on a system) http://baudizm.blogsome.com/2008/12/01/quick-tip-use-dpkg-to-replicate-between-ubuntu-installs[See this article] or briefly: on the old machine: -------------------------------------------------------------------- dpkg --get-selections > pkginstalled -------------------------------------------------------------------- on the new machine: -------------------------------------------------------------------- $ dpkg --set-selections < pkginstalled $ apt-get dselect-upgrade -------------------------------------------------------------------- If you get a bunch of warnings like: 'package blabla not in database' You'll need to install and use the 'dselect' package to set things right: -------------------------------------------------------------------- $ sudo apt-get install dselect $ sudo dselect -> Update -> Install -------------------------------------------------------------------- ==== List info about installed RPMs http://www.cyberciti.biz/faq/howto-list-installed-rpm-package/[Nice summary] -------------------------------------------------------------------- 'rpm -qa' # lists all the rpms installed, for example. -------------------------------------------------------------------- ==== List files in a .rpm ----------------------------------------------------------- rpm -qlp yaddayadda.rpm # list the files in a specific rpm rpm -ql package-name # list the files an already installed rpm ----------------------------------------------------------- ==== Unpack a RPMs ----------------------------------------------------------- # this will unpack the rpm into a tree rooted at the cwd. rpm2cpio thetarget.rpm | cpio -idmv ----------------------------------------------------------- ==== Rebuild the RPM database ----------------------------------------------------------- yum clean all rpm --rebuild ----------------------------------------------------------- ==== Repair a broken YUM database ---------------------------------------------------------------------------- yum clean metadata yum clean dbcache yum update # or even yum clean all ---------------------------------------------------------------------------- ==== dpkg Cheatsheet http://www.cyberciti.biz/howto/question/linux/dpkg-cheat-sheet.php[Nice Summary] generate a list of all the installed packages in a way that they can be reinstalled or verified post install. --------------------------------------------------------------------------- dpkg --get-selections |grep -v deinstall | cut -f1 --------------------------------------------------------------------------- ==== Force a package to be installed, even if it conflicts with an existing package. Sometimes even apt-get f*cks up. To unf*ck it, sometimes you have to force things a bit. Once the 'apt-get -f install' command has failed in all its forms and you have no more fingernails to gnaw off, this may be useful ---------------------------------------------------------- dpkg -i --force-overwrite /var/cache/apt/archives/whatever-deb-is-causing-the-problem......deb ---------------------------------------------------------- This admittedly crude approach is invaluable once the above almost-always-works 'apt-get -f install' fails. BUT, not to be taken lightly. ==== Create a USB from an ISO with 7z ------------------------------------------------------------------------- 7z x name-of-iso.iso -o/path/to/mounted/USB/drive ------------------------------------------------------------------------- see http://goo.gl/F3bk0m[this Ubuntu page for more detail.] ==== Correct the 'not a comR binary' error booting from a USB Mount the USB again on the PC. Then execute the following lines as root: ------------------------------------------------------------------------- cp -r /usr/lib/syslinux/vesamenu.c32 /syslinux/ syslinux /dev/sdb1 # or whatever the device partition is. ------------------------------------------------------------------------- ==== ECC Memory errors (from the http://moo/~hjm/clusterfork/[clusterfork] docs) this requires that the http://en.wikipedia.org/wiki/Error_detection_and_correction[EDAC system] is activated, the kernel module is inserted correctly and that the logging is working correctly. On CentOS (>5.5) and the later Ubuntu releases (>= 10.04), it appears to be. ---------------------------------------------------------------------------- cd /sys/devices/system/edac/mc && grep [0-9]* mc*/csrow*/[cu]e_count ---------------------------------------------------------------------------- strace ^^^^^^ Very useful utility to find out what an application is doing. ie *strace -p [PID of process]* See https://blogs.oracle.com/ksplice/entry/strace_the_sysadmin_s_microscope[this page for some good examples]. To strace the children threads of a parent, try this: ---------------------------------------------------------------------------- ps -efL|grep |less # Trace child processes as they are created by currently traced processes as # a result of the fork(2) system call. strace -f -p ---------------------------------------------------------------------------- and then find the child PIDs from that if you need them. http://meenakshi02.wordpress.com/2011/02/02/strace-hanging-at-futex/[From here] Updating /etc/init.d scripts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ---------------------------------------------------------------------------- sudo update-rc.d defaults # ie sudo update-rc.d sgeexecd defaults ---------------------------------------------------------------------------- Filesystems, RAIDs, and Disks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ==== Maintaining timestamps & ownership of tarchives When tarchiving old dirs, it's useful to maintain the ownership and timestamp for storage management. To do this with a dir called 'freesurfer' ------------------------------------------------------------------- $ ls -ltd freesurfer # note the date and ownership drwxr-xr-x 32 small braincircuits 59 Apr 7 2016 freesurfer # get the current timestamp and owner of the dir TS=`stat --printf=%y freesurfer` OWNER=`ls -l freesurfer | scut -f='2 3' --od='.'` # tarchive the directory as root tar -czf freesurfer.tar.gz freesurfer # Note that the creator (root) owns the tarchive and the timestamp # is the time of the creation (now) $ ls -l freesurfer.tar.gz -rw-r--r-- 1 root.root 92860763662 Mar 25 08:56 freesurfer.tar.gz # So rechown it: $ chown $OWNER freesurfer.tar.gz # and touch it to the original date: $ touch -d"${TS}" freesurfer.tar.gz $ ls -lh freesurfer.tar.gz -rw-r--r-- 1 small braincircuits 87G Apr 7 2016 freesurfer.tar.gz ------------------------------------------------------------------- ==== ulimit & open files http://stackoverflow.com/questions/12079087/fork-retry-resource-temporarily-unavailable[[See this page] --------------------------------------------------------------- $ sysctl fs.file-nr fs.file-nr = 5504 0 26301393 # where the #s above mean: # # what a users file descriptor limit is $ ulimit -Hn 8192 # how many file descriptors are in use by a user $ lsof -u 2>/dev/null | wc -l 2876 --------------------------------------------------------------- ==== BeeGFS commands (from dfs-3-1) BeeGFS GUI: Password: admin java -jar /opt/beegfs/beegfs-admon-gui/beegfs-admon-gui.jar BeeGFS Usefull Commands: beegfs-check-servers beegfs-ctl --listnodes --nodetype=metadata --details beegfs-ctl --listnodes --nodetype=storage --details beegfs-ctl --listnodes --nodetype=client --details beegfs-ctl --listnodes --nodetype=management --details beegfs-ctl --listtargets --nodetype=storage --state beegfs-ctl --listtargets --nodetype=meta --state Optimization: /data/system-files/dfs3-optimize.sh ZFS Useful commands: zpool status zfs get compressratio zpool get all | grep autoreplace List Drives byUUID and Serial /data/system-files/dfs3-list-drives-by-uuid-serial.sh ==== remount specific BeeGFS ie for /dfs3 ------------------------------------------------- service beegfs-client stop dfs3 service beegfs-helperd stop dfs3 service beegfs-helperd start dfs3 service beegfs-client start dfs3 # or simply service beegfs-client restart dfs3 service beegfs-helperd restart dfs3 ------------------------------------------------- ==== ZFS commands See https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/[this link] for a very good description of ZFS on Debian Linux. The commands blow are largely taken from that doc. - *zpool create 'poolname' 'devices'* creates a simple RAID0 of the devices named. ie: 'zpool create tank sde sdf sdg sdh' (note that you don't have to use the full device name). And You shouldn't use raw device names anyway, but the disk UIDs which are found in '/dev/disk/by-id' - *zpool status '[poolname]'* will dump the status of that pool. If the 'poolname' is omitted, 'zpool' will dump the status of all the pools it knows about. - *zfs get all [poolname]/[folder] | grep compressratio* will dump the compression ratios for the pool mentioned (http://www.sotechdesign.com.au/category/zfs/[see here]) - *zfs get all* will dump everything it knows about the ZFS pools, disk, EVERYTHING. pipe it into 'less'. - *zfs set mountpoint=/foo_mount data* will make zfs mount your data pool to a designated foo_mount point of your choice. - *zpool events* will dump all the 'events' that it has detected. //==== Some notes on disks, ECC errors, replacement, timeouts. //See also http://smartmontools.sourceforge.net/badblockhowto.html[the smartmontools // badblocks page on this], and the // http://sourceforge.net/apps/trac/smartmontools/wiki/FAQ[smartmontools FAQ]. //When disks are manufactured these days, they are equipped with a certain number of spare sectors that can be put into service when some error (usually ECC) make one sector unreliable. //In the 3TB disks that we're using for our raids, fdisk says: //------------------------------------------------------------------- //Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes //255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors //Units = sectors of 1 * 512 = 512 bytes //Sector size (logical/physical): 512 bytes / 4096 bytes //I/O size (minimum/optimal): 4096 bytes / 4096 bytes //------------------------------------------------------------------- //That implies that of the reported 5860533168 sectors of 512 bytes each = 3.00059298202e+12, all of them are dedicated to storage and none are spare. //However, ==== mdadm stopping RAIDS In order for 'mdadm' to stop a RAID, the RAID needs to be unused by other processes. Even if it appears to be untouched by local processes (via 'lsof' or 'fuser'), if the FS is NFS-exported, it can still be locked by remote processes even tho they are not immediately associated with the FS. 'mdadm' will complain that it can't stop the RAID: ------------------------------------------------------------------------ [root@compute-3-9 ~]# mdadm --stop /dev/md0 mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running process, mounted filesystem or active volume group? ------------------------------------------------------------------------ and 'fuser' will list ie ------------------------------------------------------------------------ [root@compute-3-9 compute-3-9]# fuser -m /compute-3-9 /compute-3-9: 1rce 2rc 3rc 4rc 5rc 6rc 7rc 8rc 9rc 10rc 11rc 12rc 13rc 14rc 15rc 16rc 17rc 18rc 19rc 20rc 21rc 22rc 23rc 24rc 25rc 26rc 27rc 28rc 29rc 30rc 31rc 32rc 33rc 34rc 35rc 36rc 37rc 38rc 39rc 40rc 41rc 42rc 43rc 44rc 45rc 46rc 47rc 48rc 49rc 50rc ... ------------------------------------------------------------------------ So you need to stop the NFS service as well as as killing off all the other processes (or letting them finish) with th emore polite 'umount -l /dev/md0' (umounts the FS but lets the current open files close 'naturally') Once you stop the NFS services: ------------------------------------------------------------------------ [root@compute-3-9 ~]# /etc/init.d/nfs stop Shutting down NFS daemon: [ OK ] Shutting down NFS mountd: [ OK ] Shutting down NFS quotas: [ OK ] Shutting down NFS services: [ OK ] Shutting down RPC idmapd: [ OK ] [root@compute-3-9 ~]# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 ------------------------------------------------------------------------ mdadm can stop the raid (as long as there are no more processes accessing it.) ==== deleting the mdadm RAID info from the disks If you want to use the disks from one mdadm raid in another, you'll have to 'blank them' 1st by removing the superblock info. ie, if your raid reports itself as: ------------------------------------------------------------------------ $ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md127 : active (auto-read-only) raid5 sdc1[0] sdf1[4] sde1[2] sdd1[1] 8790790656 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: ------------------------------------------------------------------------ then you'll have to stop it first (described in more detail above): ------------------------------------------------------------------------ Fri May 22 15:42:38 root@pbs2:~ 247 $ mdadm --stop /dev/md127 mdadm: stopped /dev/md127 ------------------------------------------------------------------------ and finally erase the superblock info: ------------------------------------------------------------------------ $ for II in sdc1 sdf1 sde1 sdd1; do mdadm --zero-superblock /dev/${II}; done # all gone. ------------------------------------------------------------------------ Now you can remove the disks and re-use them in another system. ==== replacing a disk in an mdadm RAID *DO NOT JUST PULL THE BAD DISK OUT* If you do, see below. The correct way to replace a disk in a mdadm RAID is to: . *mdadm fail* the disk . *mdadm remove* the disk . *only then* physically remove the disk (if you have a hotswap backplane) or . power down the system and *THEN* remove the disk, then power the system back up . add the disk back into the RAID. In this case, it's /dev/sdc, from both /proc/mdstat and dmesg: ----------------------------------------------------------- cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdf[5] sdd[2] sdc[1](F) sde[6] sdb[0] 11721064448 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [5/4] [U_UUU] # note that '[U_UUU]' - that shows that the second disk in the RAID has died. # in this case, it's /dev/sdc, from both the postional info in the [U_UUU] # and from the dmesg output, which looks like this: ... end_request: I/O error, dev sdc, sector 5439271384 end_request: I/O error, dev sdc, sector 5439271384 md/raid:md0: read error not correctable (sector 5439271384 on sdc). sd 3:0:0:0: [sdc] Unhandled sense code sd 3:0:0:0: [sdc] sd 3:0:0:0: [sdc] sd 3:0:0:0: [sdc] sd 3:0:0:0: [sdc] CDB: end_request: I/O error, dev sdc, sector 5439272072 end_request: I/O error, dev sdc, sector 5439272072 md/raid:md0: read error not correctable (sector 5439272072 on sdc). ----------------------------------------------------------- The bad disk can also be identified by close inspection of the LEDs if they're connected to the activity monitor pins: ls -lR /raid/mount/point Will cause all the disks into 'blinking/read' activity, *EXCEPT the bad disk*. Now that we know which disk, we can replace it. The 1st step is to fail it. ----------------------------------------------------------- # the bad disk is /dev/sdc (the whole disk; not just a partition) # and in this case, we have a hotswap backplane BADDISK=/dev/sdc # for ease of reference # the 1st step is to 'mdadm fail' it. mdadm --manage /dev/md0 --fail $BADDISK mdadm: set /dev/sdc faulty in /dev/md0 # then 'mdadm remove' it mdadm --manage /dev/md0 --remove $BADDISK mdadm: hot removed /dev/sdc from /dev/md0 # ONLY THEN, physically pull the disk. Once it has been replaced with a disk # of AT LEAST the same size, and the disk has spun up and been detected by the OS # 'mdadm add' the new disk (which we still refer to as $BADDISK) # mdadm --manage /dev/md0 --add $BADDISK mdadm: added /dev/sdc # then check with /proc/mdstat again cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdc[7] sdf[5] sdd[2] sde[6] sdb[0] 11721064448 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [5/4] [U_UUU] [==>..................] recovery = 11.1% (326363808/2930266112) finish=438.4min speed=98984K/sec ----------------------------------------------------------- If you drop a disk, *make sure to fail the failed disk BEFORE you physically remove it*. However if you don't do that, you can still recover, in a very nervous way: (In the following example, /dev/sdb went bad and I stupidly removed it without failing it out of the RAID6 1st. That means that mdadm lost track of 'sdb' and I wasn't able to add the replacement back in. When I tried to, I got: ------------------------------------------------------------------------ $ mdadm --manage /dev/md0 --add /dev/sdc mdadm: add new device failed for /dev/sdb as 6: Invalid argument ------------------------------------------------------------------------ In order to repair the RAID6 now, you have to re-create the RAID and THEN add the previously failed disk back in. * stop the mdadm raid: *mdadm -S /dev/md0* * re-create the RAID: ------------------------------------------------------------------------ # note the 'missing' value in the create line which acts as a placeholder mdadm --create /dev/md0 --assume-clean --level=6 --verbose --raid-devices=6 \ /dev/sda missing /dev/sdc /dev/sdd /dev/sde /dev/sdf # ^^^^^^^ mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: /dev/sda appears to be part of a raid array: level=raid6 devices=6 ctime=Thu Apr 16 10:33:32 2015 mdadm: /dev/sdc appears to be part of a raid array: level=raid6 devices=6 ctime=Thu Apr 16 10:33:32 2015 mdadm: /dev/sdd appears to be part of a raid array: level=raid6 devices=6 ctime=Thu Apr 16 10:33:32 2015 mdadm: /dev/sde appears to be part of a raid array: level=raid6 devices=6 ctime=Thu Apr 16 10:33:32 2015 mdadm: /dev/sdf appears to be part of a raid array: level=raid6 devices=6 ctime=Thu Apr 16 10:33:32 2015 mdadm: size set to 2930135040K mdadm: automatically enabling write-intent bitmap on large array Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. ------------------------------------------------------------------------ * then ad the 'new' /dev/sdb back: ------------------------------------------------------------------------ [root@compute-7-1 /]# mdadm --add /dev/md0 /dev/sdb mdadm: added /dev/sdb [root@compute-7-1 /]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdb[6] sdf[5] sde[4] sdd[3] sdc[2] sda[0] 11720540160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [6/5] [U_UUUU] [>....................] recovery = 0.0% (562176/2930135040) finish=1129.0min speed=43244K/sec bitmap: 0/22 pages [0KB], 65536KB chunk ------------------------------------------------------------------------ * now it can be mounted and used as it's rebuilding. See: ==== Wiping disks Truly wiping disks to prevent the NSA from recovering yur data is fairly pointless since they have so many other avenues to that data. However, it you want to pass on a laptop or sell an old disk on ebay without allowing your private info to be easily recovered, you can try 'dd' or 'badblocks' as described below: 'dd' without a 'count' parameter will just keep going until it hits the end of the disk so the time to finish will be proportional to the size of the disk and the speed at which it can be forced to write (modern disks write at about 100MB/s so a 3TB disk will take ~ 8.3hr to wipe the entire disk this way. Older disk of the older laptop size will be smaller (say 80GB) and slower (say 50MB/s), so such a disk will take about an hour. ------------------------------------------------------------------------ dd bs=1M if=/dev/zero of=/dev/sd# or dd bs=1M if=/dev/urandom of=/dev/sd# or badblocks -wvs /dev/sd# ------------------------------------------------------------------------ See http://superuser.com/questions/86952/disk-wiping-other-than-dban[this superuser thread] for a longer discussion and more hints. ==== Remount running filesystems to change options Sometimes it's useful to change options when a filesystem is running. This is possible and safe with modern Liux systems. --------------------------------------------------------------- mount -o remount,rw,noatime,nodiratime,swalloc,largeio,barrier,sunit=512,swidth=8192,allocsize=32m,inode64 /raid2 --------------------------------------------------------------- examining /etc/mtab should show you that the option has changed. ==== Re-setup Fraunhofer on HPC nodes Need to reset the env variables 'HPC_CURRENT_KERNEL_RPM' *ON THE NODE THAT you're trying to fix* --------------------------------------------------------------- export HPC_CURRENT_KERNEL_RPM=/data/node-setup/node-files/rpms/kernel/whatever_it_is # like this: # export HPC_CURRENT_KERNEL_RPM=/data/node-setup/node-files/rpms/kernel/kernel-2.6.32-358.18.1.el6.x86_64.rpm # and then... /data/node-setup/add-fhgfs.sh # this is pretty robust. --------------------------------------------------------------- ==== Mounting/remounting multiple Fhgfs mounts From Adam Brenner: ---------------------------------------------------------------------------- to restart /fast-scratch we would use: service fhgfs-helperd restart fast-scratch service fhgfs-client restart fast-scratch Like wise, for /dfs1 (distributed filesystem 1) service fhgfs-helperd restart dfs1 service fhgfs-client restart dfs1 ---------------------------------------------------------------------------- ==== Remove LSI 'Foreign' info When the disk that you're trying to use has some info designating it as part of another array or somehow non-native, you have to clear the 'foreign' info. Not easy to find, but the storcli does have a command for it. ---------------------------------------------------------------------------- # on controller 0, for all foreign disks, clear the foreign info ./storcli /c0/fall del ---------------------------------------------------------------------------- ==== Dump all relevant SMART info to a file ---------------------------------------------------------------------------- # then grep out the useful bits egrep '== /dev'\|'^Model'\|'^Device Model'\|'^Serial'\|'^Firm'\|'5 Reall'\|'9 Power'\|'187 Rep'\|'197 Cur'\|'198 Off'\|'199 UDMA'\|'^ATA Error'\|'Extended offline' # or with simpler plain grep: grep '== /dev\|^Model\|^Device Model\|^Serial\|^Firm\|5 Reall\|9 Power\|187 Rep\|197 Cur\|198 Off\|199 UDMA\|^ATA Error\|Extended offline' ---------------------------------------------------------------------------- ==== Remove 3ware RAID info from a disk (this should also work with disks from LSI controllers - testing now) This will remove the 3ware RAID info so the disk won't show up as being part of a previous RAID. This is quite disconcerting when a supposedly newly tested disk comes up as a failed RAID. This has been verified at least 10x by me as well. See http://www.finnie.org/2010/06/07/howto-delete-a-3ware-dcb/[this entry] In order to have direct access to the disk, need to pull it and place into another computer, easiest is another compute node, but can also do it on a desktop with the appropriate controller (dumb controllers are best; HW controllers will tend to interpose their own overhead and presentation to the OS). If the OS presents the test disk directly in the form of '/dev/sdX', that's good. ---------------------------------------------------------------------------- DISK=sdX # where X is the device of interest. Prob not 'a' COUNT=2048 # indicator of how much overwrite you want to do on begin & end LBAS=$(cat /sys/block/$DISK/size) echo "LBAS = $LBAS" dd if=/dev/zero of=/dev/$DISK bs=512 count=$COUNT dd if=/dev/zero of=/dev/$DISK bs=512 seek=$(($LBAS-$COUNT)) count=$COUNT ---------------------------------------------------------------------------- [[smart9750]] ==== Pull SMART data: 3ware 9750 *Note that this only works on SATA disks on the 9750, not SAS disks.* If the 3DM2 interface shows 'ECC errors' on a SATA disk, or other hard to figure out error, you can view the SMART info on a per-disk basis from behind the 3ware 9759 using the 'smartctl' CLI. This will allow you to check the severity of the error. ie the 3DM2 interface will tell you that there is an 'ECC error' but will not tell you whether this has resulted in uncorrectable reallocs (BAD), or whether the ECCs trigggered a correction - a valid realloc (not uncommon). You don't want to see hundreds of reallocs, especially if they're increasing in #, but having a low and stable number of reallocs is acceptable (depending on your budget and your definition of 'acceptable'). Otherwise, in order to check the error state of the disk, you have to pull the disk to examine it on another machine, causing a degraded array, and a several-hour rebuild. The 'smartctl' man pages does not make this very clear, but the sequences of 'devices' that you have to query is not the '/dev/twl[#]' but the '-d 3ware,start-finish'. ie you use the same '/dev/twl0' and iterate over the '3ware,start-finish' as shown below. In my case, the active numbers are 8-43 in a 36-slot Supermicro chassis. Yours may be anything, but I would suggest starting at 0 and going as high as you need to catch the beginning sequence. I included the '(seq 7 44)' to see if any of the chassis' started numbering outside of that range. In my case they didn't. The grep filter just grabs the bits I'm interested in. You'll have to look at the 'smartctl' output to decide what you want. You can see what gets grabbed in the example output below. [source,bash] ------------------------------------------------------------------------- $ for II in $(seq 7 44); do \ echo "== $II =="; \ smartctl -a -d 3ware,${II} /dev/twl0 | \ egrep 'Device Model'\|'Serial Number'\|'^ 5 '\|'^ 9 '\|'^187'\|'^198'\|'^199'; \ done ------------------------------------------------------------------------- And output is... ------------------------------------------------------------------------- .... (much deleted) == 42 == Device Model: ST3000DM001-9YN166 Serial Number: W1F0BZCA 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 14271 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 == 43 == Device Model: ST3000DM001-9YN166 Serial Number: W1F0A83L 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 14271 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 == 44 == ... (no output, showing that the sequence stops at 43) ------------------------------------------------------------------------- [[Falcon]] ==== Pull SMART data: LSI SAS2008 Falcon This is one of the best controllers for ZFS, which has an 'lspci' string like this: 03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) In order to extract the SMART data from this controller, you have to use the /dev/sgX syntax: ------------------------------------------------------------------------- # this takes about 45s to run on a 36 disk chassis rm -f diskscan; for ii in $(seq 0 39); do echo "== /dev/sg${ii} ==" >> diskscan; smartctl -iHA -l error -l selftest /dev/sg${ii} >> diskscan; done ------------------------------------------------------------------------- for ii in b c d; do echo >> diskscan; smartctl -iHA -l error -l selftest /dev/sd${ii} | \ egrep '== /dev'\|'^Model'\|'^Device Model'\|'^Serial'\|'^Firm'\|'5 Reall'\|'9 Power'\|'187 Rep'\|'197 Cur'\|'198 Off'\|'199 UDMA'\|'^ATA Error'\|'Extended offline' >> diskscan; done [[smartmegaraid]] ==== Pull SMART data: LSI MegaRAID Similarly, the smartmontools can pull a more limited amount of info from SAS disks connected to an LSI MegaRAID controller. In the following, the controller is an LSI Nytro 81004i with a total of 36 Hitachi SAS disks in 2 RAIDs of 17 disks each with 1 Global Hot Spare and 1 spare disk. The disks themselves can be probed with an iteration (similar to the 3ware controllers above) starting from 6 and going to 43 for some reason on this (36bay Supermicro) chassis. An example output is shown below. -------------------------------------------------------------------- # for SAS disks $ smartctl -a -d megaraid,6 /dev/sdb # for SATA disks '-d sat+megaraid,6' - 'megaraid' by itself isn't sufficient. # where the ',6' is the disk slot number # depending on the chassis and how it's wired, this number can start at 6,7,or # 8 and go as high as the number of slots + the initial offset. # the '/dev/sdb' is the device that the RAID controller presents to the OS. # can also invoke tests on the SAS disks as well as the SATA disks with: # smartctl -t short -d megaraid,24 /dev/sdb # (note that the '/dev/sdb' can also be /dev/sda in the above, if the disks # are presented as 2 arrays. It just needs to be able to find the right controller # and either of the 2 will point to it. smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.5.1.el6.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net Vendor: HITACHI Product: HUS723030ALS640 Revision: A350 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Logical block size: 512 bytes Logical Unit id: 0x5000cca03e5f8fa4 Serial number: YVHPK6LK Device type: disk Transport protocol: SAS Local Time is: Tue Feb 18 12:50:18 2014 PST Device supports SMART and is Enabled Temperature Warning Enabled SMART Health Status: OK Current Drive Temperature: 32 C Drive Trip Temperature: 85 C Manufactured in week 48 of year 2012 Specified cycle count over device lifetime: 50000 Accumulated start-stop cycles: 17 Specified load-unload count over device lifetime: 600000 Accumulated load-unload cycles: 325 Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 688410477985792 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 55798 0 55798 6350629 140344.648 0 write: 0 93714 0 93714 16256 10074.763 0 verify: 0 20 0 20 1258 1.040 0 Non-medium error count: 0 No self-tests have been logged Long (extended) Self Test duration: 27182 seconds [453.0 minutes] -------------------------------------------------------------------- Note that this doesn't have nearly the same amount of status info as it can pull from SATA disks. An example script to pull all the interesting data from an array (ie /dev/sdb) would be: -------------------------------------------------------------------- RAID="/dev/sdb" for II in $(seq 6 43); do echo "" echo "" echo "==== Slot $II ====" smartctl -a -d megaraid,${II} $RAID | egrep 'Vendor'\|'Product'\|'Serial number' done or as a 'one-liner': RAID="/dev/sdb";for II in $(seq 6 43); do echo ""; echo ""; echo "==== Slot $II ====";\ smartctl -a -d megaraid,${II} $RAID; done # pipe output thru egrep "ID1|ID2" ID3|etc" to filter the ones you want as above. -------------------------------------------------------------------- As an aside, the bash expression {a..z} does the same thing for characters. -------------------------------------------------------------------- $ echo {a..g} a b c d e f g $ for II in {j..o}; do echo $II; done j k l m n o -------------------------------------------------------------------- ==== rsync trailing '/s' There's a quirk with rsync that is not a bug, but a required finesse. Most Linux utilities don't care if you include a trailing "/". rsync does and it makes a difference both in the source and target. f you append the "/" to the source, but NOT the target, it tells rsync to sync the individual CONTENTS of the source to the target. -------------------------------------------------------------------- rsync -av nco-4.2.6 moo:~/tnco -------------------------------------------------------------------- results in what you usually expect: -------------------------------------------------------------------- Tue May 17 10:29:14 [0.11 0.12 0.13] hjm@moo:~/tnco 509 $ ls nco-4.2.6/ Tue May 17 10:30:02 [0.29 0.15 0.15] hjm@moo:~/tnco 512 $ ls -w 70 nco-4.2.6/ COPYING acinclude.m4 bld/ configure.eg m4/ INSTALL aclocal.m4 bm/ configure.in man/ Makefile.am autobld/ config.h.in data/ qt/ Makefile.in autogen.sh* configure* doc/ src/ -------------------------------------------------------------------- BUT if you append the trailing "/" to the source, you get the CONTENTS of the source synced to the target: -------------------------------------------------------------------- Tue May 17 10:31:31 [0.17 0.17 0.16] hjm@moo:~/tnco 516 $ ls -w 70 COPYING acinclude.m4 bld/ configure.eg m4/ INSTALL aclocal.m4 bm/ configure.in man/ Makefile.am autobld/ config.h.in data/ qt/ Makefile.in autogen.sh* configure* doc/ src/ -------------------------------------------------------------------- If you append a trailing "/" to the target, makes no difference. -------------------------------------------------------------------- rsync -av nco-4.2.6 moo:~/tnco # and rsync -av nco-4.2.6 moo:~/tnco/ -------------------------------------------------------------------- will both result in the result immediately above. However, -------------------------------------------------------------------- rsync -av nco-4.2.6 moo:~/tnco # followed by rsync -av nco-4.2.6/ moo:~/tnco -------------------------------------------------------------------- will result in a double syncing, with one set of files in the target dir and the second in a subdir named 'moo:~/tnco/nco-4.2.6'. so be careful. You generally want the format WITHOUT any "/"s. ==== rsync in/exclude patterns http://goo.gl/bFsJaG[This is a good overview] in understandable English. ==== Quickly delete bazillions of files with rsync the following rsync command will recursively delete files about 10X faster than the usual 'rm -rf'. See http://www.slashroot.in/which-is-the-fastest-method-to-delete-files-in-linux[this link] to read about it in more detail. It works. -------------------------------------------------------------------- mkdir empty rsync -a --delete empty/ targetdir/ # note that both dirs have a '/' at the end of the dir name. THIS IS REQUIRED # the above command will leave the top level dir, so follow with rm -rf targetdir/ -------------------------------------------------------------------- ==== And even quicker with Perl Same link as above. -------------------------------------------------------------------- perl -e 'for(<*>){((stat)[9]<(unlink))}' -------------------------------------------------------------------- Careful - this microscript is recursive and non-interactive. ==== Testing an mdadm RAID to check that the mail notification is working -------------------------------------------------------------------- sudo mdadm --monitor --scan -1 -m 'hmangala@uci.edu' -t # should have the email in the /etc/mdadm/mdadm.conf file already tho. -------------------------------------------------------------------- ==== Primitive daily mdadm email notification This section has been expanded into http://moo.nac.uci.edu/~hjm/bduc/HOWTO_verify_RAID_by_crontab.html[an entire doc], covering a number of controllers with proprietary software as well as the Linux software RAID system http://en.wikipedia.org/wiki/Mdadm[mdadm]. crontab -l: -------------------------------------------------------------------- 07 12 * * * cat /proc/mdstat| mutt -s 'PBS1 mdadm check' hmangala@uci.edu -------------------------------------------------------------------- ==== More sophisticated way of using email/mdadm checking -------------------------------------------------------------------- # this is the query command which gives a nice overview of the RAID /sbin/mdadm -Q --detail /dev/mdX # this is the entry for a crontab which would send an email that has an easy # to read subject line with details in the email body: # (all in one line for a contab entry) 4 6,20 * * * SUB=`/sbin/mdadm -Q --detail /dev/md0 |grep 'State:'`; \ /sbin/mdadm -Q --detail /dev/md0 | mutt -s "DUST RAID: $SUB" hmangala@uci.edu # if the node has multiple mdadm RAIDs, you can do all at once with: 05 6,20 * * * SUB=/sbin/mdadm -Q --detail /dev/md0 /dev/md1 \ /dev/md2 |grep 'State :' | cut -f2 -d':' | tr '\n' ' ' ; /sbin/mdadm -Q \ --detail /dev/md0 /dev/md1 /dev/md2 | mutt -s "BDUC-LOGIN MDADM RAIDs: \ $SUB" hmangala@uci.edu -------------------------------------------------------------------- ==== Assembling a pre-existing RAID on a new OS. The problem is that you had a RAID running perfectly on another OS and the system disk dies or had to be replaced for other reasons, or you needed to upgrade it with a bare-metal replacement. How do you resucitate a pre-existing RAID? It's very difficult: -------------------------------------------------------------------- $ mdadm -A [the-old-array-name] ie: $ mdadm -A /dev/md0 mdadm: /dev/md0 has been started with 4 drives. # mdadm queries the system, finds the disks, examines the embedded config info and restarts the RAID as before. $ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb1[0] sde1[4] sdd1[2] sdc1[1] 8790400512 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] -------------------------------------------------------------------- mdadm is *VERY* smart. ==== Prepping an old mdadm array for a new array you have to stop the old array before you can re-use the disks. -------------------------------------------------------------------- mdadm --stop /dev/md_d0 (or whatever the old array is called) -------------------------------------------------------------------- and then you can re-use the disks, altho you may have to force mdadm to continue to include them into the new array, like this (from 'man mdadm'): -------------------------------------------------------------------- mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[ac]1 # Create /dev/md0 as a RAID1 array consisting of /dev/hda1 and /dev/hdc1. echo 'DEVICE /dev/hd*[0-9] /dev/sd*[0-9]' > mdadm.conf mdadm --detail --scan >> mdadm.conf -------------------------------------------------------------------- The above will create a prototype config file that describes currently active arrays that are known to be made from partitions of IDE or SCSI drives. This file should be reviewed before being used as it may contain unwanted detail. ==== Replace a disk from an mdadm RAIDX http://www.clan-elite.info/forum.asp?action=view_thread&id=904[Here's how]. briefly: remove the bad partition (in the below example, sdc1), from the RAID: -------------------------------------------------------------------- mdadm --manage /dev/md0 -r /dev/sdc1 -------------------------------------------------------------------- Then power-off the machine if it doesn't have hot-swap slots and replace the disk (careful about which one it is); always a good idea to test what you think is the failed disk with a USB IDE/SATA cable set. Then power on the machine (mdadm should flash a console message that the raid is operating in a degraded state) and once it's up, format the new disk like the old disk The URL above says you can do it with: -------------------------------------------------------------------- sfdisk -d /dev/sda | sfdisk /dev/sdc -------------------------------------------------------------------- where '/dev/sda' is one of the pre-existing disks and '/dev/sdc' is the new disk. see 'man sfdisk' If you're going to use fdisk or cfdisk, use disk type 'FD' for 'Linux RAID AUTODETECT'), and re-add the disk to the RAID with: -------------------------------------------------------------------- mdadm --manage /dev/md0 -a /dev/sdc1 -------------------------------------------------------------------- Look at '/proc/mdstat' to check the rebuild status. Find the Disk UUIDs for specifying disks in /etc/fstab ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *Use blkid*. See below. While a pain and not intuitive (like /dev/sdc3 is intuitive?), using the disk UUIDs will prevent disk order swapping and possible data loss when reformatting the disk you thought was '/dev/sdb' and on reboot, turned into '/dev/sda'. ie: --------------------------------------------------------------------------- $ sudo ssh a64-182 Last login: Tue Oct 9 09:55:45 2012 from bduc-login.bduc [root@a64-182 ~]# blkid /dev/sda: UUID="d274d4cf-9b09-4ed2-a66d-7c568be7ea45" TYPE="xfs" /dev/sdb1: LABEL="/boot" UUID="510f25d0-abee-4cb1-8f1e-e7bccc37d79b" /SEC_TYPE="ext2" TYPE="ext3" /dev/sdb2: LABEL="SWAP-sda2" TYPE="swap" /dev/sdb3: LABEL="/" UUID="d0d4bc25-3e48-4ee6-8119-9ce54079ee83" /SEC_TYPE="ext2" TYPE="ext3" --------------------------------------------------------------------------- ==== Label your disks correctly in Ubuntu/Kubuntu from http://ubuntuforums.org/showthread.php?t=283131[original source] To list your devices, first put connect your USB device (it does not need to be mounted). By volume label: ----------------------------------------------------------------------------- ls /dev/disk/by-label -lah ----------------------------------------------------------------------------- By id: ----------------------------------------------------------------------------- ls /dev/disk/by-id -lah ----------------------------------------------------------------------------- By uuid: ----------------------------------------------------------------------------- ls /dev/disk/by-uuid -lah ----------------------------------------------------------------------------- IMO, LABEL is easiest to use as you can set a label and it is human readable. The format to use instead of the device name in the fstab file is: LABEL=