Run cuffdiff in a computer cluster

[satp]

Hi all,

I tried to use cuffdiff to deal with many assembled RNA-Seq data, but it fail due to the insufficient memory problem. The cuffdiff was run on a cluster, but someone tell me that it can only use the memory resource of one node, although the total memory of the cluster is sufficient for the cuffdiff. I'm not familiar with parallel computation, I want to know is there any way to make cuffdiff can use the memory resources of different nodes in a cluster?

Thanks for any help.
Jian-You Liao

[mbblack]

Hi all,

I tried to use cuffdiff to deal with many assembled RNA-Seq data, but it fail due to the insufficient memory problem. The cuffdiff was run on a cluster, but someone tell me that it can only use the memory resource of one node, although the total memory of the cluster is sufficient for the cuffdiff. I'm not familiar with parallel computation, I want to know is there any way to make cuffdiff can use the memory resources of different nodes in a cluster?

Thanks for any help.
Jian-You Liao

Cannot run it on multiple nodes for the simple reason that the only way to get accurate FDR values is to analyze all the genes at once. So if you split it out across multiple nodes, you'd never get accurate estimates of significance.

The only thing you can do is increase the available memory on that one node (and use the -p switch to use multithreading, if that node has multiple compute cores).

One way in Linux to do this is use a temporary disc file to expand swap memory space. You need root to do this, so you may need to get your cluster admin to help.

----example commands to make a 24Gb file and append it to swap----------
# dd if=/dev/zero of=/directory_name/tmpswap_file_name bs=1024 count=25165824
# mkswap -f /directory_name/tmpswap_file_name
# swapon /directory_name/tmpswap_file_name

once no longer needed

# swapoff /directory_name/tmpswap_file_name
-------------------------------------------------------------------------