"Too many open files" when running SVDetect

[Kaas]

Dear all

I have sequenced a CHO cell line descended from CHO K1 and I would like to investigate the chromosomal rearrangements that as occurred and decided upon SVDetect. I sequenced the genome to a theoretical depth of 38x

1. I made a list of the chromosomes and their length (called sortedlist.txt)
1 NW_003613580.1 8905210
2 NW_003613581.1 8197018
3 NW_003613582.1 6761507
…
4392 NW_003717424.1 213

2. I aligned my reads to the CHO K1 genome using BWA, realigned using GATK and removed duplicates using Picard.

3. I used the SVDetect BAM_preprocessingPairs script to preprocess my bam file
perl ~/genome/SVDetect_r0.8b/scripts/BAM_preprocessingPairs.pl /novo/omicsmanager/processed01/1099/data/pipe2del/DXB1/4Picard.bam

Gave the output:
Total : 62889376 pairs analysed
-- 251915 pairs whose one or both reads are unmapped
-- 62637461 mapped pairs
---- 746850 abnormal mapped pairs
------ 542994 pairs mapped on two different chromosomes
------ 443784 pairs with incorrect strand orientation and/or pair order
------ 179904 pairs with incorrect insert size distance
---- 61890611 correct mapped pairs

4. I converted the bam to sam (as needed for SVdetect input)
samtools view -h -o 4Picard.ab.sam 4Picard.ab.bam

5. I made a config file
<general>
input_format=sam
sv_type=all
mates_orientation=FR
read1_length=86
read2_length=86
mates_file=/novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam
cmap_file=/novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/sortedlist.txt
num_threads=1
</general>

<detection>
split_mate_file=1
window_size=4000
step_length=1000
</detection>

<filtering>
split_link_file=0
nb_pairs_threshold=3
strand_filtering=1
</filtering>

<bed>
<colorcode>
255,0,0=1,4
0,255,0=5,10
0,0,255=11,100000
</colorcode>

6. Finally I ran SVDetect
perl ~/genome/SVDetect_r0.8b/bin/SVDetect linking -conf /novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/configfile.txt

it gave the output:
# Splitting the mate file "/novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam" for parallel processing...
/novo/users/csrk/genome/SVDetect_r0.8b/bin/SVDetect: can't open /novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam:Too many open files

The reason is probably that there are 4392 contigs in my genome. Anyone knows of a way to get around this? or know of another piece of software they can recommend for deducing genomic rearrangements

[GenoMax]

"too many open files" indicates that you may be hitting a limit set for your account by the system administrators. The command to check the limits is "limit" or "ulimit" depending on flavor of unix you are running. You should check with your system administrators.

Representative solutions pages are here: https://rtcamp.com/tutorials/linux/i...n-files-limit/

http://blog.thecodingmachine.com/content/solving-too-many-open-files-exception-red5-or-any-other-application

[Kaas]

Dear GenoMax

Thanks for the reply. I did not know i operated under such constraints.

My "ulimit" is "unlimited"
and the /etc/security/limits.conf file is:
#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4
genedata soft nofile 65536
genedata hard nofile 65536

using the other link you sent
cat /proc/sys/fs/file-nr
69888 0 210054287

where 210,054,287 is maximum number of file descriptors for the whole system = the system does not seem to be the limiting factor... SVDetect must be tackling the problem in a weird way

but thanks a lot for bringing this to my attention

[GenoMax]

Perl has its own file handle limits (http://www.perlmonks.org/?node_id=849608). Probably do not apply here.

Error above seems to indicate that the file is being split for parallel processing but you have number of threads set to only 1 in your config file.

Are there any additional error messages beyond what you have posted above (on system side)? Are you running this under a queuing system or directly in a shell?

[Kaas]

I have run it directly in a shell (so no queuing issues should be present). I tried changing the number of threads but it made no difference. same error
I am on a very basic level when it comes to unix so i would not know to look for additional error messages beyond the one i posted above (well SVdetect started out by making an output for all the contigs in the genome and then the error message as stated above)

[GenoMax]

The error also says "can't open /novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam"

The file is there and does have read permissions?

[Kaas]

head /novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam
works just fine so the file is there and the program should be able to acess it (tried chmod a+x /novo/omicsmanager/processed01/1099/data/pipe2del/SVdetect/4Picard.ab.sam and it did not change anything).

[GenoMax]

You probably should have done (a+r rather than a+x) but otherwise we seem to have checked most obvious things.

Consider contacting the authors with the error message to see if they have any suggestions.

[Kaas]

Thanks. Great to hear that I did not just overlook something extremely basic. I have written them and will post the answer in here when they write me back

[Kaas]

FYI to anyone out there with the same problem I had: I wrote the author of SVdetect and he told me that the software was not made to handle more than 255 chromosomes for the reference. you can get around this by manually divide your sam file into individual files (4Picard.ab.sam.all.scaffoldname)

these files have to be found in the mates folder where you run your analysis
cat sortedlist.txt | gawk '{print $2}' > chrlist
wc -l chrlist > length
length=`awk '{print $1 }' < length`
for (( i = 1 ; i <= $length ; i++ ))
do
cat chrlist | head -$i | tail -1 > scaffold
scaffold=`awk '{print $1 }' < scaffold`
grep "$scaffold" 4Picard.ab.sam > ./mates/4Picard.ab.sam.all.$scaffold
done

Then i was able to run a successful analysis using SVdetect with the change in the config file from above that

<detection>
split_mate_file=0
window_size=4000
step_length=1000
</detection>

meaning that the file has already been split.

[GenoMax]

Thanks for checking back and documenting the solution for others who may chance upon this thread via search in future.