By default BBDuk only tries to grab about 42% of physical memory. You can override the amount of RAM BBDuk uses with the -Xmx flag. And you can increase the efficiency of memory usage (as well as indexing speed) with the "prealloc" flag. For example:
bbduk.sh -Xmx100g prealloc (other arguments)
Note that you should not go all the way to 128g, though the exact amount you can use before disk-swapping or refusal to initialize depends on the system configuration. For our 128g machines, it's about 105g, but I think on most systems 120g should be fine.
The memory usage is proportional to the reference, and specifically, "hdist=1" will increase the memory requirements by a factor of 3*K, which is pretty immense when you have a large reference. It roughly needs 15b per kmer at a minimum when you preallocate (up to 45 if you don't). You can also adjust memory usage by playing with the "skip" flag.
In a little while I will release a new version that has some additional features for controlling memory use and speed.
bbduk.sh -Xmx100g prealloc (other arguments)
Note that you should not go all the way to 128g, though the exact amount you can use before disk-swapping or refusal to initialize depends on the system configuration. For our 128g machines, it's about 105g, but I think on most systems 120g should be fine.
The memory usage is proportional to the reference, and specifically, "hdist=1" will increase the memory requirements by a factor of 3*K, which is pretty immense when you have a large reference. It roughly needs 15b per kmer at a minimum when you preallocate (up to 45 if you don't). You can also adjust memory usage by playing with the "skip" flag.
In a little while I will release a new version that has some additional features for controlling memory use and speed.
Comment