[GenoMax]

Are you the only user on this machine when you try to run STAR? Are there any other processes going on in background? Have you looked at output of say "top"? What does the memory section up at the top say?

[emixaM]

No problem:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 95162
max locked memory       (kbytes, -l) 4096000
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 95162
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[GenoMax]

Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 31550092110

@emixaM: Can you post the output from?

$ ulimit -a

[emixaM]

Thank you very much Alex!

I tried, for good measure, to launch STAR from the beginning, not directly on the 2nd pass, on my 24GB machine, and I got this error.

Here is the command:

STAR --runMode alignReads \
  --genomeDir Homo_sapiens.GRCh38/genome/star_index/Ensembl77.sjdbOverhang99 \
  --readFilesIn \
    sample_L001.trim.pair1.fastq.gz \
    sample_L001.trim.pair2.fastq.gz \
  --runThreadN 2 \
  --readFilesCommand zcat \
  --outStd Log \
  --outSAMunmapped Within \
  --outSAMtype BAM Unsorted \
  --outFileNamePrefix alignment_1stPass/sample/ \
  --outSAMattrRGline ID:"sample_L001" 	PL:"ILLUMINA" 	 PU:"CXX_1" 		SM:"sample" 	CN:"seq"  \
  --limitGenomeGenerateRAM 20000000000 \
  --limitIObufferSize 32000000 \
  --genomeSAsparseD 2

And the error:

Feb 08 10:39:30 ..... Started STAR run
Feb 08 10:39:31 ..... Loading genome

EXITING: fatal error trying to allocate genome arrays, exception thrown: std::bad_alloc
Possible cause 1: not enough RAM. Check if you have enough RAM 31550092110 bytes
Possible cause 2: not enough virtual memory allowed with ulimit. SOLUTION: run ulimit -v 31550092110

Feb 08 10:39:31 ...... FATAL ERROR, exiting

gzip: stdout: Broken pipe

gzip: stdout: Broken pipe

Is it the same problem?

[alexdobin]

Hi alexdobin

As many posts that you answered, I got a RAM problem.

The machines I use have 24GB of RAM and I am working with human data.

This is my command (I decreased genomeChrBinNbits argument, as I saw in your previous answers):

STAR --runMode genomeGenerate \
  --genomeDir test_star/reference.Merged \
  --genomeFastaFiles Homo_sapiens.GRCh38.fa \
  --runThreadN 2 \
  --genomeChrBinNbits 8 \
  --limitGenomeGenerateRAM 10000000000 \
  --sjdbFileChrStartEnd alignment_1stPass/AllSamples.SJ.out.tab \
  --limitIObufferSize 4000000000 \
  --sjdbOverhang 99

This is the log:

Feb 03 11:15:54 ..... Started STAR run
Feb 03 11:15:54 ... Starting to generate Genome files
Feb 03 11:17:11 ... starting to sort  Suffix Array. This may take a long time...
Feb 03 11:17:35 ... sorting Suffix Array chunks and saving them to disk...
Feb 03 12:37:51 ... loading chunks from disk, packing SA...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/var/spool/torque/mom_priv/jobs/1759223.moab.host.SC: line 41: 27870 Aborted                 STAR --runMode genomeGenerate --genomeDir test_star/reference.Merged --genomeFastaFiles Homo_sapiens.GRCh38.fa --runThreadN 2 --genomeChrBinNbits 8 --limitGenomeGenerateRAM 10000000000 --sjdbFileChrStartEnd alignment_1stPass/AllSamples.SJ.out.tab --limitIObufferSize 4000000000 --sjdbOverhang 99

The execution created 45 SA_* files of around 1GB.

What am I doing wrong in the parameters? I would like it to run with a ~20 GB of RAM consumption.

Thanks in advance for any help you might give me.

Hi @emixaM,

with 24GB of RAM and human genome you would need to use --genomeSAsparseD 2.
You do not need --genomeChrBinNbits 8 --limitIObufferSize 4000000000, and can increase
--limitGenomeGenerateRAM 20000000000

STAR --runMode genomeGenerate \
  --genomeDir test_star/reference.Merged \
  --genomeFastaFiles Homo_sapiens.GRCh38.fa \
  --runThreadN 2 \
  --limitGenomeGenerateRAM 20000000000 \
  --sjdbFileChrStartEnd alignment_1stPass/AllSamples.SJ.out.tab \
  --sjdbOverhang 99

Cheers
Alex

[emixaM]

Hi alexdobin

As many posts that you answered, I got a RAM problem.

The machines I use have 24GB of RAM and I am working with human data.

This is my command (I decreased genomeChrBinNbits argument, as I saw in your previous answers):

STAR --runMode genomeGenerate \
  --genomeDir test_star/reference.Merged \
  --genomeFastaFiles Homo_sapiens.GRCh38.fa \
  --runThreadN 2 \
  --genomeChrBinNbits 8 \
  --limitGenomeGenerateRAM 10000000000 \
  --sjdbFileChrStartEnd alignment_1stPass/AllSamples.SJ.out.tab \
  --limitIObufferSize 4000000000 \
  --sjdbOverhang 99

This is the log:

Feb 03 11:15:54 ..... Started STAR run
Feb 03 11:15:54 ... Starting to generate Genome files
Feb 03 11:17:11 ... starting to sort  Suffix Array. This may take a long time...
Feb 03 11:17:35 ... sorting Suffix Array chunks and saving them to disk...
Feb 03 12:37:51 ... loading chunks from disk, packing SA...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/var/spool/torque/mom_priv/jobs/1759223.moab.host.SC: line 41: 27870 Aborted                 STAR --runMode genomeGenerate --genomeDir test_star/reference.Merged --genomeFastaFiles Homo_sapiens.GRCh38.fa --runThreadN 2 --genomeChrBinNbits 8 --limitGenomeGenerateRAM 10000000000 --sjdbFileChrStartEnd alignment_1stPass/AllSamples.SJ.out.tab --limitIObufferSize 4000000000 --sjdbOverhang 99

The execution created 45 SA_* files of around 1GB.

What am I doing wrong in the parameters? I would like it to run with a ~20 GB of RAM consumption.

Thanks in advance for any help you might give me.

[alexdobin]

Hi @bastianwur,

quoting from the manual:

"If you are using a genome with a large (>5,000) number of references (chrosomes/scaolds), you may need to reduce the --genomeChrBinNbits to reduce RAM consumption. The following scaling is recommended: --genomeChrBinNbits = min(18, log2(GenomeLength/NumberOfReferences))."

In you case it would be --genomeChrBinNbits 9 .

Cheers
Alex

I'm trying to run STAR at the moment, but it's throwing me an error while generating the genome.

I simply use the standard command:
./STAR --runMode genomeGenerate --genomeDir /2tb-internal/data/STAR_test/ --genomeFastaFiles /2tb-internal/data/contig-100.without_polyA.fixed_names.fa --runThreadN 5

The output I get:

Jan 13 15:53:51 ..... Started STAR run
Jan 13 15:53:51 ... Starting to generate Genome files
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

Not sure what the problem could be.
I didn't compile STAR myself, because I don't have GCC 4.7 installed, therefore compiling doesn't work (Working on Ubuntu 12.04). Is that maybe a problem?
The "genome" is not very big, but is actually a transcriptome, so it's not very big (file is 115 mb), but a lot entries (~200.000), guess that could also be a problem...or not?
Hardware should else be sufficient, have 16 GB RAM, and can switch to a server with 256 GB if necessary.
Any hints what could be the factor causing the problem here?

Thanks .

[bastianwur]

I'm trying to run STAR at the moment, but it's throwing me an error while generating the genome.

I simply use the standard command:
./STAR --runMode genomeGenerate --genomeDir /2tb-internal/data/STAR_test/ --genomeFastaFiles /2tb-internal/data/contig-100.without_polyA.fixed_names.fa --runThreadN 5

The output I get:

Jan 13 15:53:51 ..... Started STAR run
Jan 13 15:53:51 ... Starting to generate Genome files
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

Not sure what the problem could be.
I didn't compile STAR myself, because I don't have GCC 4.7 installed, therefore compiling doesn't work (Working on Ubuntu 12.04). Is that maybe a problem?
The "genome" is not very big, but is actually a transcriptome, so it's not very big (file is 115 mb), but a lot entries (~200.000), guess that could also be a problem...or not?
Hardware should else be sufficient, have 16 GB RAM, and can switch to a server with 256 GB if necessary.
Any hints what could be the factor causing the problem here?

Thanks .

[rvann]

Hi @rvann,

the error is caused by the zero-length read sequence, STAR cannot process those.
Hopefully, cutadapt has an option to remove them - this has to be done simultaneously from read1 and read2 files to preserve the order of the reads.

Cheers
Alex

Thanks, Alex. I tried running cutadapt with the flag -m set to N=1 so that any 'zero-length' reads would be discarded (see below). STAR had no problems with these trimmed fastq files.

cutadapt -q 10 -m 1 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC -A AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGATCTCGGTGGTCGCCGTATCATT -o outpath/file_R1.fastq.gz -p outpath/file_R2.fastq.gz inpath/file_R1.fastq.gz inpath/file_R2.fastq.gz

[alexdobin]

Hi Alex,

Your software is really great! I had a quick question about the --quantMode GeneCounts function. I seem to be getting mostly noFeature hits. This is what is in my ReadsPerGene.out.tab file:

N_unmapped 3350825 3350825 3350825
N_multimapping 2233686 2233686 2233686
N_noFeature 4913585 40288551 40271442
N_ambiguous 0 0 0

My genome generation step included the annotation file, which is a gff3 file. Since it's a gff3 file, I added the paramters "--sjdbGTFfile xx.gff3 --sjdbGTFtagExonParentTranscript Parent --sjdbGTFfeatureExon exon". This is a subset of the gff3 file:


I'm fairly certain that I should not be getting ~40 million no feature hits when counting. Do you have any idea as to what the problem might be? Please let me know if any more information is needed.

Thanks!

Hi Yasser,

at the moment the best option is to convert GFF3 file into GTF file.
For instance, you can use gffread tool from Cufflinks package:
$ gffread -T annot.gff3 -o annot.gtf
It creates the gtf file with proper transcript_id and gene_id tags, which you can supply as --sjdbGTFfile without any Parent options.

Cheers
Alex

[salamay]

Hi Alex,

Your software is really great! I had a quick question about the --quantMode GeneCounts function. I seem to be getting mostly noFeature hits. This is what is in my ReadsPerGene.out.tab file:

N_unmapped 3350825 3350825 3350825
N_multimapping 2233686 2233686 2233686
N_noFeature 4913585 40288551 40271442
N_ambiguous 0 0 0

My genome generation step included the annotation file, which is a gff3 file. Since it's a gff3 file, I added the paramters "--sjdbGTFfile xx.gff3 --sjdbGTFtagExonParentTranscript Parent --sjdbGTFfeatureExon exon". This is a subset of the gff3 file:

##gff-version 3
##annot-version v1.0
scaffold_1 phytozomev10 gene 10215584 10239664 . + . ID=Thhalv10024176m.g.v1.0;Name=Thhalv10024176m.g
scaffold_1 phytozomev10 mRNA 10215584 10239664 . + . ID=Thhalv10024176m.v1.0;Name=Thhalv10024176m;pacid=20194900;longest=1;Parent=Thhalv10024176m.g.v1.0
scaffold_1 phytozomev10 exon 10215584 10215918 . + . ID=Thhalv10024176m.v1.0.exon.1;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 five_prime_UTR 10215584 10215821 . + . ID=Thhalv10024176m.v1.0.five_prime_UTR.1;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 CDS 10215822 10215918 . + 0 ID=Thhalv10024176m.v1.0.CDS.1;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 exon 10216476 10216579 . + . ID=Thhalv10024176m.v1.0.exon.2;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 CDS 10216476 10216579 . + 2 ID=Thhalv10024176m.v1.0.CDS.2;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 exon 10216865 10216999 . + . ID=Thhalv10024176m.v1.0.exon.3;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 CDS 10216865 10216999 . + 0 ID=Thhalv10024176m.v1.0.CDS.3;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 exon 10217082 10217132 . + . ID=Thhalv10024176m.v1.0.exon.4;Parent=Thhalv10024176m.v1.0;pacid=20194900
scaffold_1 phytozomev10 CDS 10217082 10217132 . + 0 ID=Thhalv10024176m.v1.0.CDS.4;Parent=Thhalv10024176m.v1.0;pacid=20194900

I'm fairly certain that I should not be getting ~40 million no feature hits when counting. Do you have any idea as to what the problem might be? Please let me know if any more information is needed.

Thanks!

[alexdobin]

Hi @gsgs

porting software designed for Linux to Windows is not an easy task.
As I mentioned, there is a serious effort to do that - I will ask if there is an ETA.
In the meantime, I could suggest the following work-arounds (in the order of increased difficulty):

1. Use Amazon or Google computing clouds. It will cost you a few dollars per run.
2. Run a virtual Linux machine on your Windows server.
3. Make you server dual-boot Windows/Linux, with a shared FAT partition to transfer data.
4. Try to compile and run STAR under cygwin Linux-like environment. This should be easier than full porting, however, I am not sure if this will work.

Cheers
Alex

[gsgs]

so, what can I do ?

---------------------------------------
Buy another computer with Linux on it, install "STAR" on it.
Whenever I have a big Windows/DOS fasta file to be aligned,
(delete the carriage returns since Linux doesn't like them ?)
copy it from the Windows/DOS HD to a micro-SD, insert it into the Linux computer,
run STAR on it, insert it into the Windows computer,
copy it back to Windows HD, insert the carriage returns
-----------------------------------------

do you sell such a Linux computer, with STAR suitably installed on micro-SD ?
Easy to use, boots from SD, aligns the fasta file fasta01.fa on it, writes
the result to fasta02.fa and shuts down.

No display, no keyboard needed, a raspberry computer ?!

[gsgs]

thanks.

In the meantime I found the manual and looked at it.
Whenever it mentioned "Windows" it meant the boxes
that may open ... so this is just not considered.

I'm not familiar with Linux/Unix, but I remember that often
I could compile similar programs with my
old GCC / DJGPP compiler on Windows.
I haven't done this since long and there are many
programs here with .h and .cpp extension to be included,
(why is it so complicated ?)
so a long list for potential problems.

I don't understand why this conversion is so difficult,
why they have no solution for this already.

I mean, it should be a small step as compared to
creating the programs and getting it to work in the first place ?!?


Currently I'm using MAFFT, the author had helped me to
get a Windows-executable and how to run it from batch.
They used fast fourier transform but it became clear to me,
that this is slow for large problems
and that there should be a faster solution by finding matching


subsequences.

[alexdobin]

what's the probability that I will be able to get a windows executable that runs on command line per cmd.exe on my computer and also runs from batch files

Hi @gsgs,

the probability is >0 as I know people who are working on Windows executable for STAR.

Cheers
Alex