Isaac2 genome index creation

[GenoMax]

Did the format of the index files change from isaac1 to isaac2?
I still have my isaac1 index files ... :-)

Even if the format hasn't changed I did not build the index last time (used Illumina's). With the assertion that isaac2 can be used for any genome I thought it would be a good idea to get that experience under my belt. Indexing job is still running (2 days, 32 cores, TB RAM). UCSC hg19 from iGenomes.

Illumina must be using all flash storage/newest multi-core xeons etc to get their index built in under a half day.

[sklages]

I thought I did for isaac1 .. though I will "re-test" this as well.
I haven't tried isaac2 yet.

Did the format of the index files change from isaac1 to isaac2?
I still have my isaac1 index files ... :-)

[GenoMax]

I thought this would be handled by -j in both isaac1/2.

2015-07-22 08:40:21     [7f15a156c7c0]    Version: iSAAC-02.15.07.16
  -j [ --jobs ] arg (=4)  Maximum number of compute threads to run in parallel

I will have to verify it again (am just trying to get the index built without overstepping limits set by my job scheduler) but even when I specified -j 1 in an earlier try, the job still spawned as many threads as there were cores on the node.

Have you successfully used the -j option in that manner?

[sklages]

Isaac1/2 are written in a way to take over entire hardware from a server/node in question (it is documented, as I remember). Unfortunately this is not something that works well in shared compute environments.

I thought this would be handled by -j in both isaac1/2.

2015-07-22 08:40:21     [7f15a156c7c0]    Version: iSAAC-02.15.07.16
  -j [ --jobs ] arg (=4)  Maximum number of compute threads to run in parallel

[GenoMax]

Sorry that it has been a pain. It generally takes us about half a day to generate, but that is hardware dependent. I need to figure out a good place to host the index files. Will try to get human posted in the next few days.

I am not complaining. Just trying to make you aware of how many of us access shared compute resources and how isaac1/2 may come across as a surprise to new users.

Isaac1/2 are written in a way to take over entire hardware from a server/node in question (it is documented, as I remember). Unfortunately this is not something that works well in shared compute environments. It is also hard to make Isaac1/2 behave with job schedulers since the original thread spawns multiple sub-jobs over time.

[skruglyak]

@semyon: Creating indexes for human genome for isaac2 is not trivial (as I am discovering). Process has been running for 24+ hours on a server and still going.

Making isaac2 indexes available as a download may also be a useful thing for human/mouse genomes.

Sorry that it has been a pain. It generally takes us about half a day to generate, but that is hardware dependent. I need to figure out a good place to host the index files. Will try to get human posted in the next few days.

[GenoMax]

@semyon: Creating indexes for human genome for isaac2 is not trivial (as I am discovering). Process has been running for 24+ hours on a server and still going.

Making isaac2 indexes available as a download may also be a useful thing for human/mouse genomes.

EDIT: This thread was created by moving several posts that were originally in (http://seqanswers.com/forums/showthread.php?t=61208) but were solely focused on Isaac2 genome index creation issues.