An IntroBDUCtion

You must use ssh, an encrypted terminal protocol. Be sure to use the -Y or -X options, if you want to view X11 graphics (see below).

On a Mac, use the Applications → Utilities → Terminal app.
On a WinPC, use the excellent putty. See also below.
On Linux, I assume that you know how to start a Terminal session with one of the bazillion terminal apps (konsole & terminator are 2 good ones).

Telnet access is NOT available. Use your UCINetID and associated password to log into the login node (bduc-login.nacs.uci.edu) via ssh.

To connect using a Mac or Linux, open the Terminal app and type:

As of June 15th, 2009, you can also ssh directly to the claw1 node for a 64bit interactive node from anywhere on campus.

Note that in order to help you debug login and other problems, the sysadmin’s public ssh keys are also added to your ~/.ssh/authorized_keys file. If you do not want this, you’re welcome to comment it out, but unless it’s active, I can’t help you with problems that require a direct login.

Occasionally you may get the error below when you try to log into BDUC (or more rarely, among the BDUC nodes):

The reason for this error is that the computer to which you’re connecting to has changed its identification key. This might be due to the mentioned man-in-the-middle attack but is far more likely to be an administrative change that has caused the BDUC node to have changed its ID. This may be due to a change in hardware, reconfiguration of the node, a reboot, an upgrade, etc.

The fix is buried in the error message itself.

Simply edit that file and delete the line referenced. When you log in again, there will be a notification that the key has been added to your known_hosts file.

Should you want to be able to log in regardless of this warning, you’ll have to edit the /etc/ssh/ssh_config file and add the 2 lines as shown below. (Macs, Linux). There are good reasons for not doing this, but it’s a convenience that many of us use. Consider it the rolling stop of ssh security.

After you do that, you’ll still get the warning (which you should investigate) but you’ll be able to log in.

If you’re using putty on Windows, you won’t be able to effect this security skip-around. Read why here.

Logging in to bduc.nacs.uci.edu will give you access to a Linux shell, (bash by default, tcsh, ksh available).

You will also have access to the resources of the BDUC via the SGE commands. The most frequently used commands for SGE will be qrsh to request an interactive node and qsub to submit a batch job. You can also check the status of various resources with the qconf command. See the SGE cheatsheet for more detail.

The login node should be considered your 1st stop in doing real work. You can copy files to and from your home directory from the login node, but you shouldn’t run any long (>10m) jobs on the login node. If you do and we notice, we’ll kill them off. To do real work, request a node from the interactive queue, like this:

We have not yet imposed disk quotas on BDUC. We encourage you to use the data storage you need, up to hundreds of GB, but we also warn you that if we detect large directories that have not been used in weeks, we retain the right to clean them out. The larger the dataset, the more scrutiny it will get. IF YOU HAVE LARGE DATASETS AND ARE NOT USING THEM, THEY MAY DISAPPEAR WITHOUT WARNING. We mean it when we say that if you generate valuable data, it is up to you to back it up elsewhere ASAP.

If you have no idea of how large your data is and how it is distributed, you can find out via the du command (disk usage).

To see the distribution of files graphically,

This will launch kdirstat which will determine the size, type and age of your files and present them in a color-coded map by size. You can then inspect and hopefully remove the ones least needed.

This is covered in more detail in the document HOWTO_move_data. There are currently a few ways to get your files to and from BDUC. The most direct, most available way is via scp. Besides the commandline scp utility bundled with all Linux and Mac machines, there are GUI clients for MacOSX and Windows, and of course, Linux. If you have large collections of files or large individual files that change only partially, you might be interested in using rsync as well.

Once you copy your data to your BDUC HOME directory, it is available to all the compute nodes via the same mount point on each, so if you need to refer to it in a SGE script, you can reference the same file in the same way on all nodes. ie: /home/hmangala/my/file will be the same file on all nodes.

Once you’ve generated some data on BDUC, you may want to keep it handy for a short time while you’re further processing it. In order to keep it both compact and accessible, BDUC supports the archivemount utility on both the login and claw1 nodes. This allows you to mount a compressed archive (tar.gz, tar.bz2, and zip archives) on a mountpoint as a fuse filesystem. You can cd into the archive, modify files in place, copy files out of the archive, or copy files into the archive. When you unmount the archive, the changes are saved into the archive. Here’s an extended article on it from Linux Mag.

Here’s an example of how to use archivemount with a 84MB data tarball (data.tar.gz) that you want to interact with.

sshfs is a utility that allows you to mount remote directories in your BDUC home dir. Since it operates in user-mode, you don’t have to be root or use sudo to use it. It’s very easy to use and you don’t have to alert us to use it..

You have to be able to ssh to the machine from which you want to exchange files, typically the desktop or laptop you’re connecting to BDUC from (ergo WinPCs cannot do this without much more effort). For MacOSX and Linux, in the example below assume I’m connecting from a laptop named ringo to the BDUC claw1 node. I have a valid BDUC login (hmangala) and my login on ringo is frodo.

sshfs works on both the login and claw1 nodes.

We have the full GNU toolchain available on both the CentOS interactive nodes and on all the Ubuntu/claw nodes, so normal compilation tools such as autoconf, automake, libtool, make, ant, gcc, g++, gfortran, gdb, ddd, java, python, R, perl, etc are available to you. We do not yet have any proprietary compilers or debuggers available (ie. the Intel or PGC compilers or the TotalView Debugger). Please let us know if there are other tools or libraries you need that aren’t available.

As of Jan 15, the GNU 4.6.2 series of compilers is available to all nodes. Set up the environment with module load gcc/4.6.2. Applications compiled with this series of compilers should be able to run on all nodes (the a64-XXX CentOS nodes, the clawX Ubuntu nodes and the nXXX Perceus nodes).

Logging on to an interactive node may be all that you need. If you want to slice & dice data interactively, either with a graphical app like MATLAB, VISIT, JMP, or clustalx, or a commandline app like nco or scut or even hybrids like gnuplot or R, you can run them from any of the interactive nodes, read, analyze and save data to your /home directory. As long as you satisfy the graphics requirements, you can view the output of the X11 graphics programs as well.

The bash shell allows an infinite amount of customization and shortcuts via scripts and the alias command. Should you wish to make use of such things (such as nu to show you the newest files in a directory or ll to show you the long ls output in human readable form), you can define them yourself by typing them at the commandline:

You can also place all your useful aliases into your ~/.bashrc file so that all of them are defined when you log in. Or separate them from the ~/.bashrc by placing them into a ~/.alias file and have it sourced from your ~/.bashrc file when you log in. That separation makes it easier to move your alias library from machine to machine.

A collection of such aliases is stashed on the login node at /usr/local/share/alias.login

In most cases, when you log out of an interactive session, the processes associated with that login will also be killed off, even if you’ve put them in the background (by appending the starting command with &). If you regularly need a process to continue after you’ve logged out, you should submit it to the SGE scheduler with qsub (see immediately below).

However, sometimes it is convenient to continue a long-running process when you have to log out (as when you have to shut down your network connection to take your laptop home). In this case, you can use the underappreciated screen program, which establishes a long-running proxy connection on the remote machine that you can detach from and then re-attach to without losing the connection. As far as the remote machine is concerned, you’ve never logged off, so your running processes aren’t killed off. When you re-establish the connection by logging in again, you can re-attach to the screen proxy and take up as if you’ve never been away.

You can also use screen as a terminal multiplexer, allowing multiple terminal sessions to be used from one login, especially useful if you’re using Windows with PuTTY that doesn’t have a multiple terminal function built into it.

For these reasons, screen by itself is a very powerful and useful utility, but it is admittedly hard to use, even with a good cheatsheet. To the rescue comes a screen wrapper called byobu which provides a much easier-to-use interface to the screen utility. byobu has been installed on all the interactive nodes on BDUC and can be started by typing:

There will a momentary screen flash as it refreshes and re-displays the login, and then the screen will look similar, except for 2 lines along the bottom that show the screen status. In the images below, the one at left is without byobu; at right is with byobu. The byobu screen shows 3 active sessions: login, claw_1, and bowtie. The graphical tabs at the bottom are part of the KDE application konsole which also supports multiplexed sessions (allowing you to multi-multiplex sessions (polyplex?))

The help screen, shown below, can always be gotten to by hitting the <F9> key, followed by the <Enter> key.

Most usefully, you can create new sessions with the F2 key, switch between them with F3/F4 and detach from the screen session with F6.

Note that you must have started a screen session before you can detach, so to make sure you’re always in a screen session, you can have it start automatically on login by changing the state of the Byobu currently launches at login flag (at bottom of screen after the 1st F9.

When you log back in after having detached, type byobu again to re-attach to all your running processes. If you set byobu to start automatically on login, there will be no need of this, of course, as it will have started.

Note that byobu is just a wrapper for screen and the native screen commands continue to work. As you become more familiar with byobu, you’ll probably find yourself using more of the native screen commands. See this very good screen cheatsheet.

SGE status codes:

Original table here.

The shell script that you submit (job_name.sh above) should be written in bash and should completely describe the job, including where the inputs and outputs are to be written (if not specified, the default is your home directory. The following is a simple shell script that defines bash as the job environment, calls date, waits 20s and then calls it again.

Note that your script has to include (usually at the end) at least one line that executes something - generally a compiled program but it could also be a Perl or Python script (which could also invoke a number of other programs). Otherwise your SGE job won’t do anything.

The batch queues have been to reorganized for clarity. They now are organized as follows:

To submit short jobs (<12hr), you can most easily not specify a Q - it will go on any batch Q. To run on a longxxx Q, either specify the estimated runtime in the submission script by including the -l h_rt parameter

(also see below)

or submit specifically to one of the long Qs.

ie:

Occasionally, a script will hiccup and put your job into an error state. This can be seen by the qstat state output:

the E (^{^}) means that the job is in an ERROR state. You can either delete the job with qdel:

or often change it’s status with the qmod command.

When you submit an SGE script, it is processed by both bash and SGE. In order to protect the SGE directives from being misinterpreted by bash, they are prefixed by #$ This prefix causes bash to ignore the rest of the line (considers it a comment), but allows SGE to process the directive correctly.

So, the rules are:

Here are some of the most frequently used

When a job starts, a number of SGE environment variables are set and are available to the job script.

Here are most of them:

The above was extracted from this useful page. For more on SGE shell scripts, see here.

For a sample SGE script that uses mpich2, see below

Oracles purchase of Sun has resulted in a major disorganization of SGE (now OGE) documentation. If a link doesn’t work, it may be because of this kerfuffle. tell me if a link doesn’t work anymore and I’ll try to fix it.

If you need to run an MPI parallel job, you can request the needed resources by Q as well by specifying the resources inside the shell script (more on this later) or externally via the -q and -pe flags (type man sge_pe on one of the BDUC nodes).

In a word, don’t. Many research domains generate or use multi-GB text files. Prime offenders are log files and High-Thruput Sequencing files such as those from Illumina. These are meant to be processed programmatically, not with an interactive editor. When you use such an editor, it typically tried to load the entire thing into memory and generates various cache files. (If you know of a text editor that handles such files without doing this, please let me know.)

Otherwise, use the utilities head, tail, grep, split, less, sed, and tr, possibly in combinations with Perl/Python to peek into such files and or change them.

grep especially is one of the most useful tools for text processing you’ll ever use.

For example, the following command starts at 2,000,000 lines into a file and stops at 2,500,000 lines and shows that range in the less pager.

In addition, please use the commandline utilities gzip/gunzip, bzip2, zip, zcat, etc instead of the ark graphical utility on such files. ark apparently tries to store everything in RAM before dumping it.

namd is a molecular dynamics application that interfaces well with VMD. Both of these are available on BDUC - see the output of the module avail command.

The qsub scripts to submit namd 2.7 jobs to the SGE Q’ing system are a bit tricky due to the way early namd is compiled - the specification of the worker nodes is provided by the charmrun executable and some complicated additional files supplied with the namd package. This means that namd2.7x is more complicated to set up and run than namd2.8x. The qsub scripts are provided separately below.

SATe is a Python wrapper around a number of Phylogenetic tools. It, along with its requisite tools (ClustalW2, MAFFT, MUSCLE, OPAL, PRANK, RAxML) are installed in the shared /usr/local/bin directory of the Claw nodes.

The test cases work with the default settings, but if you want to change any parameters, you have to edit the configuration file and feed it to run_sate.py with -c as shown below.

You can name the configuration file anything you like, but it has a specific format. Especially, do NOT try to start comments anywhere except the 1st character of a line, and then only beginning with #.

Here is a good SATe configuration file to start from.

Here is the same SATE configuration file with some bad comments (marked as such with BAD COMMENT in the offending line.)

Here is an example qsub submission script for SATe. Submit to SGE as:

R is an object-oriented language for statistical computing, like SAS (see below). It is becoming increasingly popular among both academic and commercial users to the extent that it was noted in the New York Times in early 2009. For a very simple overview with links to other, better resources, see this link

There are multiple versions of R on BDUC, and they do not all behave identically. Since we have a split cluster (most nodes (_80; 160 cores) run CentOS (RedHat-based) ; the 4 claw nodes (16 cores) run a version of Ubuntu, (Debian-based). Because of slightly different library structures and versions, some R add-ons don’t work across the subclusters, so in those situations, we concentrate on getting the standard approach working on the CentOS nodes, and provide work-arounds on the claw nodes.

The module system provides R versions 2.10.0 and 2.8.0 for all nodes. Additionally, the claw nodes provide version 2.9.0 because it is the default version. Finally, I’ve added the R development version which is automatically downloaded, compiled, and re-installed every night from the R archives. This is the VERY LATEST version, so new that it (infrequently) fails. Howeer, if you need the latest and greatest version, it’s available. To load any of these versions, inquire what the available versions are with module avail and then use the appropriate module load R/<version> to set up the paths.

For most things, everything works identically. The things that don’t usually have to do with parallel processing in R and the underlying Message Passing Interface (MPI) technology:

We have a single node-locked license for SAS 9.3 on claw1, a 4core Opteron node with 32GB RAM. While the license is for that node only, as many instances of SAS can be run as there is RAM for it.

To start SAS on claw1, first ssh directly to claw1:

This will start an X11 SAS session, opening several windows on your monitor (as long as you have an active X11 server running). If you’re connecting from Mac or Windows, please see this link.

You can use the SAS program editor (one of the windows that opens automatically), or use any other editor you want and paste or import that code into SAS. The combination of emacs and ESS (Emacs Speaks Statistics) is a very powerful combination. It’s mostly targeted to the R language, but it also supports SAS and Stata.

Nedit also has a template file for SAS.

BDUC supports several MPI variants.

MATLAB can be started from the login node by typing matlab. This will log you into a 64bit interactive node and start the MATLAB Desktop. matlabbig will start an interactive session on one of the claw nodes (32GB RAM).

We have 3 licenses for interactive MATLAB on the BDUC cluster. Those licenses are decremented from the campus MATLAB license pool. They are meant for running interactive, relatively short-term MATLAB jobs, typically less than a couple hours. If they go longer than that, or I see that you’ve launched several MATLAB jobs, they are liable to be killed off.

If you want to run long jobs using MATLAB code, the accepted practice is to compile your MATLAB .m code to a native executable using the MATLAB compiler mcc and then submitting that code, along with your data to a batch Q (see above for submitting batch jobs). This approach does not require a MATLAB license, so you can run as many instances of this compiled code for as long as you want without impacting the campus licenses.

The official mechanics of doing this is described here.

Some additional notes from someone who has done this is in the Appendix.

There are a number of MATLAB alternatives, the most popular of which are available on BDUC. Since these are Open Source, they aren’t limited in the number of simultaneous uses, altho you should always try to run batch jobs in the SGE queue if possible. See this doc for an overview of them and further links.

Hadoop is a Java-based framework for running large-grained, parallel jobs on clusters. It now encompasses a large number of subprojects, but it is usually used with the MapReduce approach. It scales very well, but it is complex to run on BDUC because it requires its own filesystem and scheduler. Since on BDUC (and other general-purpose clusters which are not dedicated to hadoop full-time) the job scheduling is more general-purpose, we have to run it as a meta-job. That is, you submit a request to SGE to allocate a number of nodes on which to run hadoop; SGE allocates them to hadoop; hadoop sets up the logical structures it needs on those allocated nodes and everyone’s happy. We run hadoop under myHadoop a small bit of middleware designed to handle the interactions between SGE and Hadoop.

Note that Hadoop initializes its own filesystem on the existing /scratch directory for each node. Because Hadoop can end up storing many GBs of data, we have set up a dedicated hadoop SGE Q named (surprise) hadoop. All the nodes in this Q should have >300GB available and you can test this by running the following command on the login node:

This will start an interative script that will create a subdir in the current directory which will contain files named for all the hadoop nodes which lists the free diskspace on /scratch. You can see the results by doing this:

An SGE submit script for Hadoop is here.

The usual way to exploit Hadoop is to write your application in Java, typically wrapping it into a jarfile. This Java requirement is not absolute tho. You can also write your application in Jython (Python written in Java) and therefore essentially write Java using Python. You can also write your hadoop app in pure Python (or even in C++). Here is a page describing that approach.

The hadoop installation can be found at $HADOOP_HOME. By referring to this variable, you can, for example, add standard libraries to the class path of your java application (e.g. Class-Path: $HADOOP_HOME/lib/hadoop-0.20.2-tools.jar).

(Thanks to Fabian Lindenberg for helping to set up and debug Hadoop on BDUC.)

Thanks to Dr. Steve Jenks, the claw8 node contains 2 Nvidia C1060 Graphics Processing Units (GPUs), each with 240 cores. These cores are specialized to do SIMD tasks very fast. For example, if your code supports that kind of processing, you can get 10-100X speedup for those parts of the code. You can learn more about programming these GPUs with CUDA at the local docs or via the more up-to-date docs at NVIDIA.

In order to use the GPUs, you will have to be registered to use the GPU SGE Queue (contact harry.mangalam@uci.edu) and of course will have to provide your own code, altho the entire Nvidia SDK examples are compiled and available locally: source code at /apps/gpu/1.0/NVIDIA_GPU_Computing_SDK/C/src/, compiled biaries at /apps/gpu/1.0/NVIDIA_GPU_Computing_SDK/C/bin/. Of course, since only claw8 has the GPUs installed, you’ll only be able to run them there.

In order to run the compiled GPU code for long runs, you’ll have to submit them thru the SGE scheduler using the gpu queue which is initialized by using the module load gpu directive in your qsub script. We expect that you’ll do your debugging on your own machine altho you can do it on claw8 after you have registered and been added to the gpu group.

We currently have the CUDA Toolkit 4.0 installed and will try to remain current with NVIDIA upgrades.

In order to have access to these X11 tools via Linux, your local Linux must have the X11 libraries available. Unless you have explicitly excluded them, all modern Linux distros include X11 runtime libraries. Don’t forget to use the the -Y flag when you connect using ssh to tunnel the X11 display back to your machine:

The MacOSX installation DVDs come with a free, Apple-certified X11 installation. On Leopard it’s in Optional Installs → Optional Installs.mpkg All you have to do is install it and start it running in the background to accept the X11 windows (Applications → Utilities → X11) Ditto the -Y ssh flag as above.

There are quite a few ways to use a Linux system besides logging into it directly from the console.

We’ve added GPL’ed NXservers to the claw1 and claw6 nodes. Both have direct external connections. With the nxclient software installed on your machine, you can run the entire KDE or GNOME Desktop with remarkable speed. Here’s a screenshot of my laptop screen with the nxclient Desktop from claw1 running SAS, matlab, and tablet (a genomics assembly viewer):

Get the appropriate client software for your platform here.

A compute cluster is typically composed of a pool of computers (aka nodes) that allow users (and there are usually several to several hundred simultaneous users) to spread compute jobs over them in a way that allows the maximum number of jobs to matched to number of computers. The cluster is often composed of specialized login nodes, compute nodes, storage nodes, and specialty nodes (ie: a large memory node, a GPU node, an FPGA node, a database server node, etc)

The BDUC cluster consists of about 100 computers, each of which has 2-64 64bit CPU cores and 8-256GB RAM. All these nodes have a small amount of local storage (filesystems (fs) that are directly connected with the node that hold its Operating System, a few utilities and some scratch space (in /scratch). Some nodes have considerably larger local storage to provide more storage for a specific application or to the research group that bought it. All the nodes communicate with each other over a private 1 Gb/s ethernet network, via a few central switches. This means that each node can communucate at almost 100MB/s total bandwidth with all the other nodes but there is a bottleneck at the switches and at frequently used nodes, such as the login node and at main storage nodes.

In BDUC, the login node and a main storage are unfortunately the same one, leading to a bandwidth and CPU bottleneck at the login node. This is one main reason to consider staging data elsewhere (on the remote node /scratch storage or on the /gl filesystem) for jobs that require large amounts of data I/O. See the note on staging data above.

The CPU bottleneck is important because servicing disk read and write requests requires CPU attention in the same way that an application does. BDUC’s 100-odd compute nodes are busy consuming or producing data and quite a lot of the input and output is directed at the login node’s storage. So if someone is trying to run a compute-intensive application on the login node (especially a multicore application) it leaves very little CPU time to handle the disk requests. The Operating System is fairly robust in that it will queue such requests quite deep in order to buffer them but if the CPU cannot keep up with the requests, eventually it will run out of queue depth and the system will lock up.

SO, DO NOT RUN APPLICATIONS ON THE LOGIN NODE.

It’s fine to edit your program or data files (as long as they’re under a few MB), and even compile your programs on the login node. You can also test your programs as long as they run for very short periods - a few minutes at most. If you need to run your programs interactively, request an interactive node via SGE or ssh directly to one of them (a64-00[12], claw[1-5]).

As noted above, the main storage system for BDUC was the /home filesystem, provided by the bduc-login node. The /home filesystem is a RAID6 of 15x1TB disks. RAID6 means that it can lose 2 disks before it will lose any data. However, if more than 2 disks are lost, ALL data will be lost. It has been supplemented by the /gl filesystem which is a distributed gluster fs. On the gluster fs, data is spread file-wise over 8 RAID6s on 4 different servers, each of which hosts 1/4 of the files, so even if a whole node is destroyed, 3/4 of the files will survive. This may be of little comfort if the one file you need is among the lost, but that’s why we repeat the mantra Back up your files if they are of value.

The Strongly Suggested approach is to put your code and and small intermediate analyses on /home and keep your large data and intermediate files on /gl. In this way, you’ll be able to search thru your files quickly, but when you submit large jobs to the cluster via SGE, they won’t bog down the login node, nor will they interfere with other cluster jobs since /gl is a distributed FS (and on average, only 1/4 of the cluster nodes will be interacting with the same storage node as your job). In other words, it scales well.

Unlike the Network File System (NFS) provided by /home, the /gl filesystem (fs) has some odd quirks that can either destroy performance or be used to greatly enhance it. If you have written your own code or are using an app that writes zillions of tiny chunks of data to STDOUT, and you are storing the results on the /gl fs, you should consider passing the output thru gzip to consolidate the writes into a continuous stream. If you don’t do this, each write will be considered a separate IO event and the performance on the /gl fs will be .. well .. atrocious. We recently ran into this with the bedtools utility genomeCoverageBed. It read in a multi-GB file quickly but then choked while writing the ~2GB output file, increasing the runtime by >30X.

If, however, the STDOUT is passed thru gzip, the wallclock runtime decreases even below the usual runtime and you end up with an output file that it already compressed to about 1/5 the usual size.

The here’s how to do it using the explicit example we used and a generic one

Note that the gzip trick mentioned above will only work with applications that write their data to STDOUT. If the app explicitly writes to a file, then you can use a named pipe (aka fifo - first in, first out) to capture and process the output.

To new users, especially to users who have never done BIG DATA work before: Understand what it is you’re trying to do and what that means to the system. Consider the size of your data, the pipes that you’re trying to force it thru and what analyses you’re trying to get it to perform.

It should not be be necessary to posit this, but there are clearly users who don’t understand it. There is a 1000 fold difference between each of these:

BDUC has about 30TB of storage on /gl to be shared among 400 users, and the instantaneous needs of those users varies tremendously. We do not use disk quotas to enforce user limits to allow substantial dynamic storage use. However, if you use hundreds of GB, the onus is on you to clean up your files and decrease that usage as soon as you’re done with it.

This subject - arcane as it might seem - is important enough to merit its own subsection. Because BDUC is community infrastructure, efficient use of its resources is important. Let me try this analogy: driving a car across country in 1st gear is possible, but it’s not efficient, wastes time and energy, and will decrease the life of the engine. Similarly, using Zillions Of Tiny files (ZOTfiles) is a similarly destructive practice. A tiny file by itself is no more inefficient than a huge one. If you have only 100bytes to store, store it in single file. However, the problems start compounding when there are many of them. Because of the way data is stored on disk, 10 MB stored in ZOTfiles of 100bytes each can easily take up NOT 10MB, but more than 400MB - 40 times more space. Worse, data stored in this manner makes many operations very slow - instead of looking up 1 directory entry, the OS has to look up 100,000. This means 100,000 times more disk head movement, with a concommittent decrease in performance and disk lifetime. If you are writing your own utilities, whether in Perl , C, Java, or Haskell, please use efficient data storage techniques, minimally as indexed file appending, preferably as real data storage such as binary formats, HDF5 and netCDF, and don’t forget about in-memory data compression (for example, using the excellent free zlib library or language-specific libraries that use compression, such as:

If you are using someone else’s analytical tools and you find they are writing ZOTfiles, ask them, plead with them to fix this problem. Despite the sophistication of the routines that may be in the tools, it is a mark of a poor programmer to continue this practice.

Passwordless ssh will allow you to ssh/scp to frequently used hosts without entering a passphrase each time. The process below works on Linux and Mac only. Windows clients can do it as well, but it’s a different procedure. However, regardless of your desktop machine, you can use passwordless ssh to log in to all the nodes of the BDUC cluster once you’ve logged into the login node.

In a terminal on your Mac or Linux machine, type:

save to the default places.

For the BDUC cluster case: Since all cluster nodes share a common /home, all you have to do is rename the public key file (normally id_rsa.pub in your ~/.ssh dir) to authorized_keys.

For unrelated (non-cluster) hosts: Linux users, use the ssh-copy-id command, included as part of your ssh distribution. (Mac users will have to do it manually, described just below.) ssh-copy-id does all the copying one shot, using your ~/.ssh/id_rsa.pub key (by default; use the -i option to specify another identity file, say ~/.ssh/id_dsa.pub if you’re using DSA keys)

What this does is to scp id_rsa.pub to the remote host (the ssh server your’re trying to log into) and append that key to the remote file ~/.ssh/authorized_keys. If things don’t work, check that the id_rsa.pub file has been appended correctly.

Then verify that it’s worked by ssh’ing to BDUC. You shouldn’t have to enter a password anymore.

For Mac users, scp the same keys to the remote host and append your public key to the remote ~/.ssh/authorized_keys. Here are the commands below. Just modify the UCINETID value and mouse them into the Terminal window on your local Mac.

(Thanks to Michael Vershinin and Fan Wang for their help and patience in debugging this procedure).

As noted, the official docs for compiling your MATLAB code is is described here. Before you start hurling your .m code at the compiler, please read the following for some hints.

The following is a simple case where all the MATLAB code is in a single file, say test.m. Note that for the easiest path, you should write your MATLAB code to compile as a function. This means that keyword function has to be used to define the MATLAB code (see example below). If you want to pass parameters to the function, you have include a function parameter indicating this.

Many of the problems we hear about are due to missing or incompatible library dependencies. A complicated program (like R) has many such dependencies:

and each of them typically has more, so it’s fairly common for an update to break such dependency chains, if only due to a few missing or changed functions.

If you run into a problem that seems to related to this, such as:

The above extract implies that the library libfunky.so.2 can’t find libfrenemy.so.3 to resolve missing functions, so that lib may be missing on the node that emitted the error.

If this error is emitted from a node during a batch job, it may be hard to debug which nodes are in error. To resolve this by yourself, it’s sometimes useful to use clusterfork to debug the problem.

In the above case, you would issue a command such as:

where the libfunky.so.2 is the library in question. The results will capture the STDERR and STDOUT from the single-quoted command in node-named files in a subdir that begins with REMOTE_CMD- in the working directory. Examining those files usually identify the offending nodes.

Please be careful in using cf since you can easily overwhelm the cluster if the command demands a lot of CPU or disk activity. Try the command on one node first to determine the effect and only issue the cf command after you’ve perfected it.