CASAVA and Sun Grid Engine. Orphanaligner issues?

[caseybea]

Hi everyone! First time-poster here, hoping for some help. I've been banging my head on this for a month. I am a sysadmin.

Here's the deal: We're having issues running the CASAVA Alignment step, but only when run in a grid engine envrionment. When we run it on a standalone server (using a bunch of cores.. "make -j") - it works fine. But, we have a lot of work to process, so I really need to run this in a grid engine setup to better utilize our hardware.

When we run it in the grid engine setup, the job FINISHES, and even exits with a 0 status code, but the log file shows a few of these:

[2012-10-14 03:40:46] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 1 reads (Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa) could not be found.

[2012-10-14 03:42:01] [baku.local] [ct_Rader/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 2 reads (Temp/Project_Rader/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa) could not be found.

... which basically means the alignment steop is garbage.

The problem seems to be in the orphan alignment step-- but I'm at my wits end trying to figfure it out.

Has anyone else seen anything like this? I am open to any suggestions. I am also very desperate.

Background useful tidbits:

* When run on a single node, works fine. Just takes... forever.
* ISILON high performance storage for everything
* Dedicated 10-gig network underneath
* all nodes running RedHat 5, CASAVA 1.8.2.
* SUN Grid Engine, but have also tried the open source forks. Same.
* Other CASAVA grid engine steps work fine in the cluster, for example, Variant Calling.

Whoever helps solve this gets a nice package featuring Milwaukee treats sent to their front door. Note, we make AWESOME beer here....

ILLUMINA has been less than helpful, mostly because we didn't buy "their" Illuminacompute (cluster), even though we have all the same components. It's frustrating.

[GenoMax]

Re-reading your post the question below may be the only important thing but here goes.

One question and one suggestion.

1. How is the storage mounted on compute nodes? Are those mounts permanent?

2. We used "distmake" instead of normal make (http://distmake.sourceforge.net/pmwiki/pmwiki.php). Yes it has not been updated since 2006, but it works with SGE.

[caseybea]

Re-reading your post the question below may be the only important thing but here goes.

One question and one suggestion.

1. How is the storage mounted on compute nodes? Are those mounts permanent?

2. We used "distmake" instead of normal make (http://distmake.sourceforge.net/pmwiki/pmwiki.php). Yes it has not been updated since 2006, but it works with SGE.

1. Yes, the storage is indeed mounted permanently on the compute nodes.

2. We're actually using "qmake" which comes with grid engine. It's also what Illumina is using as well as I understand other sites.


I will however, check out distmake. At this juncture, I'll look at anything.

[GenoMax]

1. Yes, the storage is indeed mounted permanently on the compute nodes.

So there should be no issues with mounts. I wonder if the disk caching is causing the problem.

2. We're actually using "qmake" which comes with grid engine. It's also what Illumina is using as well as I understand other sites.

I will however, check out distmake. At this juncture, I'll look at anything.

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

[caseybea]

So there should be no issues with mounts. I wonder if the disk caching is causing the problem.

Offhand, I don't think so. With much work between me and the ISILON people, I have eliminated any sort of caching as an issue here.

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

That's awesome. Can you perhaps post here how you're submitting your CASAVA job(s) via distmake? (specifically, alignment, since that's what I'm fighting).

[caseybea]

One addition, which is interesting to note: With the data set I'm fiddling with (and re-aligning, and re-aligning, and..) - the two files it claims as missing (and yes, they're not there), are the SAME two each time. Exact same error. And, it's not always the same execd node that's running that particular part of the job when it borks.

With possible issues (and I've investigated them all) such as disk latency, network switch issues, etc etc, after 16 hours of execution and multiple nodes involved, I would actually expect the problem to "jump around" a bit, failing in slightly different places.

Ultimately, I personally suspect some serious flakiness in Illumina's cluster-aware alignment code.

[caseybea]

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

Can I ask how qmake was failing for you? I'm very curious.

[GenoMax]

One addition, which is interesting to note: With the data set I'm fiddling with (and re-aligning, and re-aligning, and..) - the two files it claims as missing (and yes, they're not there), are the SAME two each time. Exact same error.

Can you tell us what files are missing? There are some command line options to get around missing file problems.

Are you certain that the basecalling process is completing and that you are getting sequence files for all samples/lanes.

[GenoMax]

Can I ask how qmake was failing for you? I'm very curious.

It was three years ago so I no longer remember the exact problem. We ultimately stumbled upon "distmake" as a side effect of that investigation.

[caseybea]

My missing file(s)

This is what I see when we use qmake, and describes what is missing:

It's looking for a file with a ".oa" extension, and well, that file(s) is simply not there. It's not being created. (I verified with a KEEP_FILES_.. parameter that the files in question never show up).

Here's the relevent lines from the alignment output that show what goes missing:

[2012-10-17 05:59:57] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds



[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Reason: Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Prereqs: Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml


This chunk looks like the qmake output that is LOOKING for the .oa file(s), which is not there. (see: --ie1, --ie2) My crude understanding is that this step gets run after all the eland_ms's are finished, and it's now trying to deal with orphan alignments. That's the royal pain about this, I don't see this occur until after about 16 hours. This makes for a real long time in-between tweaks.....

[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Cmd: /usr/local/casava-1.8.2/libexec/CASAVA-1.8.2/kagu --ie1 "Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa" --sl1 "32" --ub1 "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" --oe1 "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz.tmp" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_006.fastq.gz" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_004.fastq.gz" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_013.fastq.gz" --ie2 "Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa" --sl2 "32" --ub2 "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" --oe2 "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz.tmp" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_006.fastq.gz" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_004.fastq.gz" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_013.fastq.gz" --irs "Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml" --os Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml.tmp --mmaq 4 --circular=cMT.fa 1>&2 && mv Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml.tmp Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml && mv "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz.tmp.gz" "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz" && mv "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz.tmp.gz" "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz"


This ultimately generates these errors:


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 1 reads (Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa) could not be found.


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 2 reads (Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa) could not be found.




Again, this problem only seems to happen when using qmake versus make. Yet, the folks at Illumina are supposedly running this without issues, using qmake. I sure as heck wish I knew what was DIFFERENT about their setup.

Anyway, pursuing distmake sounds like an option-- do you have an example of how I'd use it for a typical Illumina run? (qsub a distmake? example parameters?). ANY sample scripts or command examples would REALLY be appreciated.

[GenoMax]

Anyway, pursuing distmake sounds like an option-- do you have an example of how I'd use it for a typical Illumina run? (qsub a distmake? example parameters?). ANY sample scripts or command examples would REALLY be appreciated.

I am going to wing a part of this since I have not used SGE recently with CASAVA v.1.8.2. Following assumes that you have completed the "configureBclToFastq" run and have the sequence files available in "Unaligned" directory.

You would do the "configureAlignment.pl" (page 12 of CASAVA 1.8.2 quick guide) normally on the command line. Then change to the "Aligned" directory (if you use a different name then to that directory) to do the qsub.

Shell script (for tcshell, adjust accordingly for your shell) for the qsub would be something like this

#!/bin/tcsh
#$ -cwd 
#$ -q YOUR_QUEUE_NAME

#$ -pe YOUR_PARALLEL_RUN_ENVIRONMENT_NAME NO_OF_CORES (we use 12)
#$ -o distmake_sge.log

We use "modules" for software management so the following two lines load appropriate modules for CASAVA.

module purge
module load pipeline

distmake -d recursive 

echo 'all done'

[GenoMax]

A suggestion.

While you are debugging the alignment process you may want to download a small data set (any bacteria will do) from SRA and use that instead of the human data. Alignments would go through much quicker and you would not need to wait for 16 hr before a run fails.

Are you using ~2GB (10-12 million reads per file) chunks for your sequences as recommended by illumina for ELAND (what is your ELAND_FASTQ_FILES_PER_PROCESS setting)?

[caseybea]

Are you using ~2GB (10-12 million reads per file) chunks for your sequences as recommended by illumina for ELAND (what is your ELAND_FASTQ_FILES_PER_PROCESS setting)?

Ours is set at the default, 3, and based on the BCL options I see in the makefile (4,000,000), that gives is 12 million - which I understand should be ideal.

[caseybea]

I'll talk to our bioinformatics people to help get me some sample data. I am currently waiting for a regular "make" to complete again in my current data set to re-verify that I do not see any failure.

[caseybea]

OK, I have distmake set up and have a job running on 3 nodes (x 8 = 24 cores).

I am excited.

At the same time, I'm a bit panicked-- as I am worried how or if distmake is supported moving forward? I love , LOVE open source solutions, but I feel a bit more comfortable if the code is in someone's loving hands keeping current.........