[TKTKTK]

I'm not sure if the problem is running up against my computation time limit - the job I specify to the scheduler is listed as completed in much less than the time allotted, and I am only running alignment on one individual to my reference at a time.

I'm currently submitting jobs with different thread values, and in my output I receive a message indicating the number of reads processed in the initial iteration of bwa-mem, but there aren't any additional iterations of this output before I receive a list of all of the nodes in my reference assembly, and then what appears to be the sam formatted bwa mem alignment output.

One thing I am curious about is how the output for bwa mem is presented and stored. Is there normally separate output in the terminal for bwa mem's run details (# of reads processed, iteration, etc), and only the sam formatted alignment data is written into the .sam file, or is all of this information included in the .sam file specified using ">"?
In the scheduling system I'm working on, all of this information is written into a user-specified output file using "-o", and when I use the ">" command, bwa mem writes an additional blank file. All of the bwa mem run details and the sam format data appear to be written into the output file.

I'm wondering if additional reads are being processed after the first sam formatted alignment output, but I can't see it because it's buried ~400,000 lines in my output file.
Hopefully this isn't the case, but I'm wondering if there is any easy way to check?

[dpryan]

Ah. So the trick here is to write a little shell script that runs bwa and uses redirection (">") internally. You then schedule that to run and you'll find that you suddenly don't have blank SAM files.

BTW, bwa will only write the alignments to the file specified with ">" (at the moment, it's not being told to do that, since the commands you're using to schedule things are getting that).

[GenoMax]

If you are using LSF then you could enclose the entire bwa command in double quotes. The file redirect will then work.

[TKTKTK]

@dpryan - I'm fairly new to this stuff, what do you mean by getting bwa to run internally? Do you mean that I write a script with run details for bwa mem in it with the specified output, and then submit a scheduled job for that script?

@GenoMax - the scheduling environment is SQ, which is a unified frontend for LSF+RMS and Maui+Torque. I'm not sure which scheduling environment I'm actually working in beyond that, is there usually something equivalent to your double quotes solution for other schedulers?

[dpryan]

Yes, that's what I mean.

[GenoMax]

@TKTKTK: Here are some examples of job submissions under SQ: https://www.sharcnet.ca/help/index.p...mmand_to_SQ.3F. After the SQ options you can try enclosing the remainder of the bwa command in double quoutes (all the way to the end including all bwa options). This works for LSF alone and may for SQ.

There is also an example of the wrapper script that Devon was referring to at the link above (scroll down on the page).

[TKTKTK]

Thanks for everyone's advice - running a script through the scheduler worked - the specified .sam file was written correctly, and I was able to confirm that bwa mem ran multiple iterations through the log file.