[GenoMax]

You could make a "htslib" directory in your STAR source and copy that file in there.

[neokao]

I did have SAMTOOLS installed. Say under my NGS folder, I have samtools-1.2 folder and STAR-STAR_2.4.0k folder. I did STARforMac in the source directory (in STAR-STAR_2.4.0k).
Any advice? Thanks.

[GenoMax]

@neokao: You need to install the new htslib library that is part of the samtools package: http://sourceforge.net/projects/samt...iles/samtools/

[neokao]

Please try the latest STAR release https://github.com/alexdobin/STAR/re...ag/STAR_2.4.0k - I have improved the BAM sorting and it now should require less RAM. Also, it may be safer to use a separate BAM sorting limit for RAM, say --limitBAMsortRAM 10000000000 .

Cheers
Alex

I tried that particular .fq file using old STAR with SAM output and then it went through.
I still want to test your new version.
(Thanks for your new code.) However, I got error when I tried to compile it.
clang: error: no such file or directory: 'htslib/libhts.a'
make: *** [STARforMac] Error 1

(I did install gcc on my OSX Yosemite)

[alexdobin]

I guess so too. I manually did these 20 .fq files with occasional Killed: 9 error. I found that it could usually go through if I run the EXACT code again (even without rebooting the OSX). However now I really got stuck with one biggest .fq file (~ 6.6G). For that particular .fg file, I got Abort trap: 6 error at ..... Started sorting BAM step. It happens everytime (tried 6~7 times so far even with a fresh reboot). I did not see anything weird with top command.
I also tried the --limitIObufferSize 100000000 but still got the Abort trap: 6 error.
It is frustrating since this is the last file to map. The particular log.out file is attached here. Thanks for the advice.

Please try the latest STAR release https://github.com/alexdobin/STAR/re...ag/STAR_2.4.0k - I have improved the BAM sorting and it now should require less RAM. Also, it may be safer to use a separate BAM sorting limit for RAM, say --limitBAMsortRAM 10000000000 .

Cheers
Alex

[neokao]

I suspect that there is some problem with RAM management as STAR takes almost all of the available RAM.
Can you try to reboot your machine - I heard that this helps some Mac systems to "declutter" RAM?
Also, please run "top" command while running STAR to see how much memory is being used.

Cheers
Alex

I guess so too. I manually did these 20 .fq files with occasional Killed: 9 error. I found that it could usually go through if I run the EXACT code again (even without rebooting the OSX). However now I really got stuck with one biggest .fq file (~ 6.6G). For that particular .fg file, I got Abort trap: 6 error at ..... Started sorting BAM step. It happens everytime (tried 6~7 times so far even with a fresh reboot). I did not see anything weird with top command.
I also tried the --limitIObufferSize 100000000 but still got the Abort trap: 6 error.
It is frustrating since this is the last file to map. The particular log.out file is attached here. Thanks for the advice.

[alexdobin]

hello,

is it possible that mapping speed for the 2nd pass decreases to 7 M/hr if 900.000 new splice sites and a comprehensive gene model (gencode v19) were used for index generation.
first pass was ~100-fold faster (700 M/hr).

my concrete syntax:
/home/ws/SW_install/STAR/STAR/source/STAR --runThreadN 31 --outSAMstrandField intronMotif --outSAMtype BAM SortedByCoordinate --genomeDir $indices --readFilesCommand zcat --readFilesIn /home/ws/data/PatientData/$b/m*/$s/t*/date*/$f1 /home/ws/data/PatientData/$b/m*/$s/t*/date*/$f2 --outFileNamePrefix $name

wiht 31 cores on a 192 GByte Scientific Linux 7 workstation.

Can I do something to improve speed?

dietmar

Hi Dietmar,

there were some reports of the slowdown in the 2nd pass. In one of the cases it was caused by the splice junctions (likely false positive) in the mitochondrion genome: https://groups.google.com/d/msg/rna-...Y/0jSn0vy0ccgJ.
If filtering out chrM junctions does not help, please send me the list of junctions from the 1st pass and a few million reads for testing.

Cheers
Alex

[alexdobin]

Thanks, Alex.

I don't know what's causing the problem for multi-threading on my Mac mini.
I got the same error even after I rebooted the computer and started the mapping freshly.

Anyways, I went ahead and started the mapping one by one with --runThreadN 1 already.
It is weird that even with --runThreadN 1 in code like this:

> STAR --genomeDir ./GenomeDir/ --readFilesIn ./BGI_RNAseq_data_2015/01.fq --runThreadN 1

, it sometimes worked but sometimes did not.

For the same .fq file, it could give the the Killed: 9 error and when I redid with the exact same code, it went through successfully. Very strange.

My .fq files do have distinct prefixes and they are ordered by two digits number as described before: 01.fq, 02.fq, etc. Could you shed more light on --readFilesIn XX.fq --outFileNamePrefix XX ? Thanks.

You can map each of the FASTQ files in the separate directory, e.g. 01/, 02/ ... The output files in all directories will have the same names, such as 01/Aligned.out.sam 02/Aligned.out.sam ...
Alternatively, you can run all STAR jobs in one directory but with different prefixes corresponding to your FASTQ files, i.e.
STAR --readFilesIn 01.fastq --outFileNamePrefix 01_
STAR --readFilesIn 02.fastq --outFileNamePrefix 02_
In this case the output file will have the specified prefixes for each of the runs, i.e.
01_Aligned.out.sam, 02_Aligned.out.sam ...

I suspect that there is some problem with RAM management as STAR takes almost all of the available RAM.
Can you try to reboot your machine - I heard that this helps some Mac systems to "declutter" RAM?
Also, please run "top" command while running STAR to see how much memory is being used.

Cheers
Alex

[dietmar13]

2-pass speed about 7 M/hr

hello,

is it possible that mapping speed for the 2nd pass decreases to 7 M/hr if 900.000 new splice sites and a comprehensive gene model (gencode v19) were used for index generation.
first pass was ~100-fold faster (700 M/hr).

my concrete syntax:
/home/ws/SW_install/STAR/STAR/source/STAR --runThreadN 31 --outSAMstrandField intronMotif --outSAMtype BAM SortedByCoordinate --genomeDir $indices --readFilesCommand zcat --readFilesIn /home/ws/data/PatientData/$b/m*/$s/t*/date*/$f1 /home/ws/data/PatientData/$b/m*/$s/t*/date*/$f2 --outFileNamePrefix $name

wiht 31 cores on a 192 GByte Scientific Linux 7 workstation.

Can I do something to improve speed?

dietmar

[neokao]

Thanks, Alex.

I don't know what's causing the problem for multi-threading on my Mac mini.
I got the same error even after I rebooted the computer and started the mapping freshly.

Anyways, I went ahead and started the mapping one by one with --runThreadN 1 already.
It is weird that even with --runThreadN 1 in code like this:

> STAR --genomeDir ./GenomeDir/ --readFilesIn ./BGI_RNAseq_data_2015/01.fq --runThreadN 1

, it sometimes worked but sometimes did not.

For the same .fq file, it could give the the Killed: 9 error and when I redid with the exact same code, it went through successfully. Very strange.

My .fq files do have distinct prefixes and they are ordered by two digits number as described before: 01.fq, 02.fq, etc. Could you shed more light on --readFilesIn XX.fq --outFileNamePrefix XX ? Thanks.

[alexdobin]

So it went through successfully. With the code like this:
> STAR --genomeDir ./GenomeDir/ --readFilesIn ./BGI_RNAseq_data_2015/01.fq,./BGI_RNAseq_data_2015/02.fq,./BGI_RNAseq_data_2015/03.fq,./BGI_RNAseq_data_2015/04.fq,./BGI_RNAseq_data_2015/05.fq,./BGI_RNAseq_data_2015/06.fq,./BGI_RNAseq_data_2015/07.fq,./BGI_RNAseq_data_2015/08.fq,./BGI_RNAseq_data_2015/09.fq,./BGI_RNAseq_data_2015/10.fq,./BGI_RNAseq_data_2015/11.fq,./BGI_RNAseq_data_2015/12.fq,./BGI_RNAseq_data_2015/13.fq,./BGI_RNAseq_data_2015/14.fq,./BGI_RNAseq_data_2015/15.fq,./BGI_RNAseq_data_2015/16.fq,./BGI_RNAseq_data_2015/17.fq,./BGI_RNAseq_data_2015/18.fq,./BGI_RNAseq_data_2015/19.fq,./BGI_RNAseq_data_2015/20.fq --runThreadN 1
, it took ~20 hours at the speed of ~37M reads per hour (That Mac mini was in use for other stuff then).

Somehow --runThreadN 2, --runThreadN 4 or --runThreadN 6 all got the same errors.

Anyway, the output is one ~200GB Aligned.out.sam file. I thought I will get 20 individual sam files to do featurecounts. Is there anyway to parse it ? Or I need to redo the STAR mapping all over with the input fq file individually? Thanks.

If read names in each of the .fastq files have distinct prefixes, you could split the resulting SAM into separate files.
Otherwise, you would have to map each of the files separately in a cycle. Something like "--readFilesIn XX.fq --outFileNamePrefix XX" will allow storing separate output files in one directory.
It's strange that multi-threading does not work. Is it possible that other stuff takes up a significant chunk of RAM?

Cheers
Alex

[neokao]

So it went through successfully. With the code like this:
> STAR --genomeDir ./GenomeDir/ --readFilesIn ./BGI_RNAseq_data_2015/01.fq,./BGI_RNAseq_data_2015/02.fq,./BGI_RNAseq_data_2015/03.fq,./BGI_RNAseq_data_2015/04.fq,./BGI_RNAseq_data_2015/05.fq,./BGI_RNAseq_data_2015/06.fq,./BGI_RNAseq_data_2015/07.fq,./BGI_RNAseq_data_2015/08.fq,./BGI_RNAseq_data_2015/09.fq,./BGI_RNAseq_data_2015/10.fq,./BGI_RNAseq_data_2015/11.fq,./BGI_RNAseq_data_2015/12.fq,./BGI_RNAseq_data_2015/13.fq,./BGI_RNAseq_data_2015/14.fq,./BGI_RNAseq_data_2015/15.fq,./BGI_RNAseq_data_2015/16.fq,./BGI_RNAseq_data_2015/17.fq,./BGI_RNAseq_data_2015/18.fq,./BGI_RNAseq_data_2015/19.fq,./BGI_RNAseq_data_2015/20.fq --runThreadN 1
, it took ~20 hours at the speed of ~37M reads per hour (That Mac mini was in use for other stuff then).

Somehow --runThreadN 2, --runThreadN 4 or --runThreadN 6 all got the same errors.

Anyway, the output is one ~200GB Aligned.out.sam file. I thought I will get 20 individual sam files to do featurecounts. Is there anyway to parse it ? Or I need to redo the STAR mapping all over with the input fq file individually? Thanks.

[alexdobin]

Somehow I could not send private message successfully, the new log.out file is in this link https://app.box.com/s/0nqn3jknqq1ta66jl1evme8c8ldunq9k

I guess that the issue started when it was creating the 8th thread. So I changed to --runThreadN 1 and then it seems OK so far for 5 hours already (still mapping). Is it normal to take more than 5 hours for 20 files like these described in the question? Thanks for your suggestions.

You can check Log.progress.out to see how many reads have been mapped. Since you are using one thread, I would expect the speed to be ~100M reads per hour, so it could take ~8 hours for ~800M reads.
You can try --runThreadN 7 or 6, or you can reduce the per-trhead IO buffer size with --limitIObufferSize 100000000.

Cheers
Alex

[neokao]

Somehow I could not send private message successfully, the new log.out file is in this link https://app.box.com/s/0nqn3jknqq1ta66jl1evme8c8ldunq9k

I guess that the issue started when it was creating the 8th thread. So I changed to --runThreadN 1 and then it seems OK so far for 5 hours already (still mapping). Is it normal to take more than 5 hours for 20 files like these described in the question? Thanks for your suggestions.

[neokao]

Hey, Alex:

Just tried with reindexing genome.
Code:
STAR --runMode genomeGenerate --genomeDir ./GenomeDir --genomeFastaFiles ./Mus_musculus.GRCm38.dna.primary_assembly.fa --runThreadN 8 --sjdbGTFfile ./Mus_musculus.GRCm38.79.gtf --sjdbOverhang 48 --genomeSAsparseD 2 --genomeSAindexNbases 13

It was completed and then I tried the same code to map the files.

Still got the same error:
Mar 27 13:16:44 ..... Started STAR run
Mar 27 13:18:05 ..... Started mapping
Killed: 9

Any more suggestions?
Thanks.