CASAVA and Sun Grid Engine. Orphanaligner issues?

[GenoMax]

Re-reading your post the question below may be the only important thing but here goes.

One question and one suggestion.

1. How is the storage mounted on compute nodes? Are those mounts permanent?

2. We used "distmake" instead of normal make (http://distmake.sourceforge.net/pmwiki/pmwiki.php). Yes it has not been updated since 2006, but it works with SGE.

[caseybea]

Hi everyone! First time-poster here, hoping for some help. I've been banging my head on this for a month. I am a sysadmin.

Here's the deal: We're having issues running the CASAVA Alignment step, but only when run in a grid engine envrionment. When we run it on a standalone server (using a bunch of cores.. "make -j") - it works fine. But, we have a lot of work to process, so I really need to run this in a grid engine setup to better utilize our hardware.

When we run it in the grid engine setup, the job FINISHES, and even exits with a 0 status code, but the log file shows a few of these:

[2012-10-14 03:40:46] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 1 reads (Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa) could not be found.

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 2 reads (Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa) could not be found.

... which basically means the alignment steop is garbage.

The problem seems to be in the orphan alignment step-- but I'm at my wits end trying to figfure it out.

Has anyone else seen anything like this? I am open to any suggestions. I am also very desperate.

Background useful tidbits:

* When run on a single node, works fine. Just takes... forever.
* ISILON high performance storage for everything
* Dedicated 10-gig network underneath
* all nodes running RedHat 5, CASAVA 1.8.2.
* SUN Grid Engine, but have also tried the open source forks. Same.
* Other CASAVA grid engine steps work fine in the cluster, for example, Variant Calling.

Whoever helps solve this gets a nice package featuring Milwaukee treats sent to their front door. Note, we make AWESOME beer here....

ILLUMINA has been less than helpful, mostly because we didn't buy "their" Illuminacompute (cluster), even though we have all the same components. It's frustrating.