novoalign multi-threading or para-processing?

[qqcandy]

I have a question about para-processing of novoalign:

If we have 1 node with 8 cores and 8 lanes of Illuimina data to process, there are two ways to run it:

1. assign 1 lane of data to each node, so that 8 lanes will be processed simultaneously on 8 nodes independently;

2. run the 8 lanes of data sequentially using the multi-threading novoalign (-c 8);

Which one is faster, if there is a difference?

Thanks in advance !

[nilshomer]

I have a question about para-processing of novoalign:

If we have 1 node with 8 cores and 8 lanes of Illuimina data to process, there are two ways to run it:

1. assign 1 lane of data to each node, so that 8 lanes will be processed simultaneously on 8 nodes independently;

2. run the 8 lanes of data sequentially using the multi-threading novoalign (-c 8);

Which one is faster, if there is a difference?

Thanks in advance !

Use the threading. I have found that it works well and then you have to avoid a merging step.

[Torst]

1. assign 1 lane of data to each node, so that 8 lanes will be processed simultaneously on 8 nodes independently;
2. run the 8 lanes of data sequentially using the multi-threading novoalign (-c 8);
Which one is faster, if there is a difference?

I would choose Option 2. You will have 8 cores processing a single data set. This will minimize RAM and disk I/O. Option 1 would require 8x as much RAM and cause lots of disk I/O switching between inputs.

[qqcandy]

Thanks a lot! I think we've got a consensus for option 2

[sparks]

Just clarify, option 1 wouldn't use 8x RAM as the index is in a shared memory segment and common to all instances of Novoalign. Performance would be similar.
Colin

[lh3]

@sparks

Is it a feature of novoalign, or a feature of Linux? On LSF, we have to request enough memory for each job; otherwise LSF will kill the job due to memory limit.

[sparks]

Linux allows files to be mapped to memory with mmap() function and multiple programs can mmap() and share the same file. Novoalign mmap()s the index so if multiple copies of Novoalign are running and using the same index then only one copy of the index is in memory.
LSF I don't know about.

[nilshomer]

Linux allows files to be mapped to memory with mmap() function and multiple programs can mmap() and share the same file. Novoalign mmap()s the index so if multiple copies of Novoalign are running and using the same index then only one copy of the index is in memory.
LSF I don't know about.

Also check out shm.h, which allows processors to share memory by attaching a shared memory segment to their own processes memory segment.