[caseybea]

Yep, it works

OK, well I am thrilled... ECSTATIC ... to report that distmake, hooked into SGE worked flawlessly. The alignment completed like it should.

If you look into the makefiles that ship with CASAVA, I am reasonably sure that there is an issue with PairAux1.mk and PairAux.mk (look at the comments in those files, you'll see what I am referring to).

Sadly, I'm not a programmer and don't posess the knowledge to fix these. It would be awesome if someone else here knew how to address it :-)

Basically, what's going on is some sort of (for lack of a better description), "dependancy leak" that is not an issue with make or distmake, but IS an issue with qmake. (and, I tried multiple flavors of grid engine, all the same issue)

The rub is, Illumina doesn't realize nor admit this, so we're stuck with crappy code. For some reason, it works in THEIR environment. I can't explain why. There are tons of variables between how the data is sequenced, what options are used in the casava alignment step, the coverage of the data, etc etc etc.

As a side note, I found it interesting to learn they do most of their in-house alignments BY LANE (and not flowcell), submitting simple "make -j 4" jobs to their grid (and thus, AVOIDING QMAKE).

The gotcha for us at the moment is, on top of everything else, we're a CAP/CLIA certified clinical lab, and as such, we NEED to use casava because it's all that's clinically licensed so far. Maybe some day someone can figure out how to get other tool(s) like bwa certified. Until then,.....

Anyway, distmake works. I think I only need it for the alignment step, we never had issues with other casava tasks in SGE.

[GenoMax]

OK, I have distmake set up and have a job running on 3 nodes (x 8 = 24 cores).

I am excited.

At the same time, I'm a bit panicked-- as I am worried how or if distmake is supported moving forward? I love , LOVE open source solutions, but I feel a bit more comfortable if the code is in someone's loving hands keeping current.........

Hope the update on distmake is positive

We went from using distmake with SGE to LSF since the lsmake utility in LSF requires a separate license

We are going to continue having this dilemma with software provided by vendors. On one hand I can understand that Illumina is unable to support the multitude of OS configurations at each institution but on the other hand people like us have to make the software work in our environments, no matter what.

You are not alone in this so rest assured that when the next major change from Illumina comes down the pipe, many of us will be attacking that collectively to find a workable solution.

[caseybea]

OK, I have distmake set up and have a job running on 3 nodes (x 8 = 24 cores).

I am excited.

At the same time, I'm a bit panicked-- as I am worried how or if distmake is supported moving forward? I love , LOVE open source solutions, but I feel a bit more comfortable if the code is in someone's loving hands keeping current.........

[caseybea]

I'll talk to our bioinformatics people to help get me some sample data. I am currently waiting for a regular "make" to complete again in my current data set to re-verify that I do not see any failure.

[caseybea]

Are you using ~2GB (10-12 million reads per file) chunks for your sequences as recommended by illumina for ELAND (what is your ELAND_FASTQ_FILES_PER_PROCESS setting)?

Ours is set at the default, 3, and based on the BCL options I see in the makefile (4,000,000), that gives is 12 million - which I understand should be ideal.

[GenoMax]

A suggestion.

While you are debugging the alignment process you may want to download a small data set (any bacteria will do) from SRA and use that instead of the human data. Alignments would go through much quicker and you would not need to wait for 16 hr before a run fails.

Are you using ~2GB (10-12 million reads per file) chunks for your sequences as recommended by illumina for ELAND (what is your ELAND_FASTQ_FILES_PER_PROCESS setting)?

[GenoMax]

Anyway, pursuing distmake sounds like an option-- do you have an example of how I'd use it for a typical Illumina run? (qsub a distmake? example parameters?). ANY sample scripts or command examples would REALLY be appreciated.

I am going to wing a part of this since I have not used SGE recently with CASAVA v.1.8.2. Following assumes that you have completed the "configureBclToFastq" run and have the sequence files available in "Unaligned" directory.

You would do the "configureAlignment.pl" (page 12 of CASAVA 1.8.2 quick guide) normally on the command line. Then change to the "Aligned" directory (if you use a different name then to that directory) to do the qsub.

Shell script (for tcshell, adjust accordingly for your shell) for the qsub would be something like this

#!/bin/tcsh
#$ -cwd 
#$ -q YOUR_QUEUE_NAME

#$ -pe YOUR_PARALLEL_RUN_ENVIRONMENT_NAME NO_OF_CORES (we use 12)
#$ -o distmake_sge.log

We use "modules" for software management so the following two lines load appropriate modules for CASAVA.

module purge
module load pipeline

distmake -d recursive 

echo 'all done'

[caseybea]

My missing file(s)

This is what I see when we use qmake, and describes what is missing:

It's looking for a file with a ".oa" extension, and well, that file(s) is simply not there. It's not being created. (I verified with a KEEP_FILES_.. parameter that the files in question never show up).

Here's the relevent lines from the alignment output that show what goes missing:

[2012-10-17 05:59:57] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds

[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] WARNING: prereq 'Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa' is not in the file system after waiting for 75 seconds



[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Reason: Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Prereqs: Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml


This chunk looks like the qmake output that is LOOKING for the .oa file(s), which is not there. (see: --ie1, --ie2) My crude understanding is that this step gets run after all the eland_ms's are finished, and it's now trying to deal with orphan alignments. That's the royal pain about this, I don't see this occur until after about 16 hours. This makes for a real long time in-between tweaks.....

[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] q-make Cmd: /usr/local/casava-1.8.2/libexec/CASAVA-1.8.2/kagu --ie1 "Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa" --sl1 "32" --ub1 "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" --oe1 "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz.tmp" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_006.fastq.gz" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_004.fastq.gz" --if1 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_013.fastq.gz" --ie2 "Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa" --sl2 "32" --ub2 "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy" --oe2 "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz.tmp" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_006.fastq.gz" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_004.fastq.gz" --if2 "/research/Human/120112_SN206_0265_BD0M1LACXX/Unaligned_0bpmm_casava182/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_013.fastq.gz" --irs "Project_Test/Sample_205P/205P_GCCAAT_L001_genomesize.xml" --os Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml.tmp --mmaq 4 --circular=cMT.fa 1>&2 && mv Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml.tmp Project_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml && mv "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz.tmp.gz" "Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_export.txt.gz" && mv "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz.tmp.gz" "Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_export.txt.gz"


This ultimately generates these errors:


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 1 reads (Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R1_001_eland_extended.txt.oa) could not be found.


[2012-10-17 06:01:12] [baku] [ct_Test/Sample_205P/205P_GCCAAT_L001_001_pair.xml] ERROR: The ELAND extended file for the mate 2 reads (Temp/Project_Test/Sample_205P/205P_GCCAAT_L001_R2_001_eland_extended.txt.oa) could not be found.




Again, this problem only seems to happen when using qmake versus make. Yet, the folks at Illumina are supposedly running this without issues, using qmake. I sure as heck wish I knew what was DIFFERENT about their setup.

Anyway, pursuing distmake sounds like an option-- do you have an example of how I'd use it for a typical Illumina run? (qsub a distmake? example parameters?). ANY sample scripts or command examples would REALLY be appreciated.

[GenoMax]

Can I ask how qmake was failing for you? I'm very curious.

It was three years ago so I no longer remember the exact problem. We ultimately stumbled upon "distmake" as a side effect of that investigation.

[GenoMax]

One addition, which is interesting to note: With the data set I'm fiddling with (and re-aligning, and re-aligning, and..) - the two files it claims as missing (and yes, they're not there), are the SAME two each time. Exact same error.

Can you tell us what files are missing? There are some command line options to get around missing file problems.

Are you certain that the basecalling process is completing and that you are getting sequence files for all samples/lanes.

[caseybea]

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

Can I ask how qmake was failing for you? I'm very curious.

[caseybea]

One addition, which is interesting to note: With the data set I'm fiddling with (and re-aligning, and re-aligning, and..) - the two files it claims as missing (and yes, they're not there), are the SAME two each time. Exact same error. And, it's not always the same execd node that's running that particular part of the job when it borks.

With possible issues (and I've investigated them all) such as disk latency, network switch issues, etc etc, after 16 hours of execution and multiple nodes involved, I would actually expect the problem to "jump around" a bit, failing in slightly different places.

Ultimately, I personally suspect some serious flakiness in Illumina's cluster-aware alignment code.

[caseybea]

So there should be no issues with mounts. I wonder if the disk caching is causing the problem.

Offhand, I don't think so. With much work between me and the ISILON people, I have eliminated any sort of caching as an issue here.

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

That's awesome. Can you perhaps post here how you're submitting your CASAVA job(s) via distmake? (specifically, alignment, since that's what I'm fighting).

[GenoMax]

1. Yes, the storage is indeed mounted permanently on the compute nodes.

So there should be no issues with mounts. I wonder if the disk caching is causing the problem.

2. We're actually using "qmake" which comes with grid engine. It's also what Illumina is using as well as I understand other sites.

I will however, check out distmake. At this juncture, I'll look at anything.

Distmake works. We have been using it for many years. "qmake" did not work on our rocks cluster.

[caseybea]

Re-reading your post the question below may be the only important thing but here goes.

One question and one suggestion.

1. How is the storage mounted on compute nodes? Are those mounts permanent?

2. We used "distmake" instead of normal make (http://distmake.sourceforge.net/pmwiki/pmwiki.php). Yes it has not been updated since 2006, but it works with SGE.

1. Yes, the storage is indeed mounted permanently on the compute nodes.

2. We're actually using "qmake" which comes with grid engine. It's also what Illumina is using as well as I understand other sites.


I will however, check out distmake. At this juncture, I'll look at anything.