Why is the samtools multi-threaded argument undocumented?

[id0]

There are a bunch of discussion threads regarding samtools multi-threaded argument (-@). However, I can't seem to find any official documentation regarding that. Does anyone know why that is? Is it safe to use? Does it work for only a subset of samtools commands?

On a related note, are there other undocumented arguments that might be useful?

[sdriscoll]

Weird - I never knew about that option. Maybe that's what was absorbed from the psamtools project that disappeared. It appears to be an option for 'view' and 'sort'. I just tried it with 'view' and samtools does in fact show > 100% cpu use and it converted sam to bam much faster than without so...I guess it's legit. I tried it with 'sort' as well and it worked fine - much faster.

[maubp]

There is an open pull request to rename this to -p since the '@' sign causes trouble for some scripting languages like Perl,

https://github.com/samtools/samtools/pull/25

(This was opened a while ago so it may be too late in terms of breaking existing scripts)

[jkbonfield]

There are at least two different multi-threading implementations in Samtools. Heng Li's and Nils Homer's. The latter appears to be more efficient as it multi-threads decoding too, but it's less clear how to control it to just, say, 500% cpu (as it'll use the same number of threads encoding as decoding). Nils' version takes a -n parameter between "samtools" and the subcommand. Eg samtools -n 4 flagstats a.bam.

Samtools is undergoing a lot of changes at the moment though with the migration to htslib, and in time the multi-threading issues will be addressed too.

[GenoMax]

Samtools is undergoing a lot of changes at the moment though with the migration to htslib, and in time the multi-threading issues will be addressed too.

Is there an ETA for this?

[jkbonfield]

Not that I know of. I believe the first thing planned is a new official release of the reorganised code base, so some unknown length of time after that for threading investigations.

We also plan to add CRAM support too, but right now I'm still working on that as part of Staden io_lib's sam/bam/cram code.

[jkbonfield]

It's not samtools, but maybe still of interest. I released the new version of Staden Package's io_lib (aka libstaden-read in some linux distribution) which contains a multi-threaded BAM/CRAM/SAM converter called scramble.

The threading is still a work in progress and it should therefore be considered as experimental. It works well for BAM (comparable to Nils Homer's implementation), less well for CRAM (maximum maybe 6x speed up, but varies depending on data set), and is as yet only single-threaded for SAM I/O although the SAM reading and writing is far faster than in Samtools.

https://sourceforge.net/projects/staden/files/io_lib/1.13.2/

The code hasn't been tested on Windows yet. Older io_lib releases work, but almost certainly the use of pthreads has broken that. I still need to back-port it to windows, or at least the MinGW/Msys environment.

I'm NOT planning on adding things like sorting and the basic experiments at merging and pileup are really just demonstration / testing tools for myself. Fundamentally this library was written first and foremost to provide an I/O layer for Gap5 (and gap4, xgap, etc before that). Some of the code will make its way into Samtools later on though - specifically the CRAM implementation.