[jwfoley]

AGREED.
I am curious though if I were to use a SSD as a swap would I be in a sweet zone for $$ vs speed?
but I guess it's a moot question since for some reason I can't find programs that allow you to choose to write to disk or use RAM.

As others noted, this is probably done at the OS level (and the OS is probably Linux if you're building a server this powerful, so it should be easy), but an SSD is way slower than RAM.

Just remember that even the least efficient software is designed to work on someone's machine, so there's an upper limit on how much RAM you'll ever need to be able to run a program. That limit might be on the order of tens of gigabytes (I've heard 40-50 for certain well-known pipelines). But I don't think there's a reason to complement that with SSDs, because they're definitely not going to buy you any additional speed as virtual memory.

[Markiyan]

Storage perfomance notes.

PS:
SSD drives HATE random writing in small blocks (they want to write in 64-128-256) KB blocks, but are OK random reads.
Harddisks struggle with random reads/writes, and also with simultanious reading/writing in multiple threads.
Raid5 is TERRIBLE for random writing. (4-5 times slower than RAID10).
My suggestion for the high perfomance system:
(each group is on separate physical HDD or raid array)
[System+Software]
[SWAP]
[SCRATCH]
[Input data]
[output data]
or at least:
[System+Software]
[SWAP+SCRATCH]
[all data]
Remember, that having input/output data on the same RAID0 array is always slower, than having them on separate disks w/o raid
(in memory constrained situation).
If working with tons of small files (phd_dir) order them by the inode number, and then read them sequentially - will be a lot faster (10-20X) than random IO on the same HDD.
Use symlinks to faciliate data processing/organisation.
If you want to use RAID - use RAID10 on the GOOD controller (Adaptec) with at least .25-1GB of the onbord cache and it's own I/O CPU. The perfomance gains with cheap onboard controllers (w/o cache) are often negative... so use separate disks, if can't afford proper RAID.