BFAST malloc error under OS X

[davidkip]

Hi

I have been using BFAST successfully for ABI alignment to hg18 for a while now. I have fresh installs of versions 0.6.1c, 0.6.3c and the latest 0.6.4e. Each version is make by

sh autogen.sh
./configure
make

Depending which I point PATH to I find that the first two work fine under Mac OS 10.5.8 on a lowly Duocore/4Gb machine, but the most recent version falls over when making a reference index from an exons files.

So

bfast index -f hg18all.fa -A 1 -n 2 -m 1111111111111111111111 -w 14 -t -i 1 -x exonFile.txt -T /tmp/

... runs fine for the first two versions, but with 0.6.4e gives a malloc error (terminal output pasted below). Interestingly, the 0.6.4e version does run successfully on my Snow Leopard laptop, suggesting some sensitivity to the background OS.

Best regards

David

=========


dk$ bfast index -f $hg18dir/hg18all.fa -A 1 -n 2 -m 1111111111111111111111 -w 14 -t -i 1 -x $exondir/pan73regions.txt -T /tmp/
************************************************************
Checking input parameters supplied by the user ...
Validating fastaFileName /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa.
Validating exonsFileName /Users/dk/Desktop/pan/documentation/pan73regions.txt.
Validating tmpDir path /tmp/.
Input arguments look good!
************************************************************
************************************************************
Printing Program Parameters:
programMode: [ExecuteProgram]
fastaFileName: /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa
space: [Color Space]
mask: 1111111111111111111111
depth: 0
hashWidth: 14
indexNumber: 1
repeatMasker: [Not Using]
startContig: 0
startPos: 0
endContig: 2147483647
endPos: 2147483647
exonsFileName: /Users/dk/Desktop/pan/documentation/pan73regions.txt
numThreads: 2
tmpDir: /tmp/
timing: [Using]
************************************************************
Reading in exons from /Users/dk/Desktop/pan/documentation/pan73regions.txt.
Read in 73 exons.
************************************************************
Reading in reference genome from /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa.cs.brg.
In total read 24 contigs for a total of 3080419480 bases
************************************************************
Creating the index...
************************************************************
Warning: startContig was less than zero.
Defaulting to contig=1 and position=1.
************************************************************
************************************************************
Warning: endContig was greater than the number of contigs in the reference genome.
Defaulting to reference genome's end contig=24 and position=57772954.
************************************************************
Currently on [contig,pos]:
[------24,---57772954]
Sorting by thread...
Merging sorts from threads...
Out of 1 required merges, currently on:
1
Merge complete.
Sorted.
bfast(20212,0xa00e8720) malloc: *** mmap(size=1073741824) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
************************************************************
In function "RGIndexCreateHash": Fatal Error[MallocMemory]. Variable/Value: index->starts.
Message: Could not allocate memory.
***** Exiting due to errors *****
************************************************************

[nilshomer]

Looks like you ran out of memory, how much do you have in the failing and successful machines respectively?

[davidkip]

4Gb in failed machine, 8Gb in successful one. The index (when made) is reported as being 1.09Gb in size (~86Mb file). Watching the memory usage shows BFAST using around 1.3Gb of memory (and there still being some free) at the time of crash.

The exons file covers ~ 14.5Mb of the human genome.

Curiously, the 0.6.3c version on the 4Gb machine works fine with no memory issues. Indeed, I've been able to make some very large indexes in this environment (eg all of hg18, split into 16 subindexes).

[nilshomer]

There is no difference in the indexing portion between the versions, so I am at a loss.

[davidkip]

Is fasta2brg identical as well (as the dependent earlier step)?

Anything I can do to help debug this?

[davidkip]

Update:

Following on from your comment that the indexing is identical in the different versions, I made an index under an earlier version and then used that as the input to the rest of the workflow (now switched to the more update BFAST).

Result: bfast match falls over with a malloc error as well.

This sounds like a general compilation difference between the two versions?

After doing

sh autogen.sg && ./configure

on freshly unpacked tarballs of each version, I did a file difference to see the difference between the makefiles (see below). Apart from a few lines where the version numbers change, a removal of the solid2fastq.pl script in the current release, there are two main differences:

91c91
< CFLAGS = -Wall -g -O2 -pthread
---
> CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64
110c110
< LIBS = -lz -lm -lbz2
---
> LIBS = -lbz2 -lz -lz -lm

Does this give any clues as to what is going on?

All the best

David

===============



$ diff /Users/dk/Desktop/ngs/bfast-0.6.4e/Makefile /Users/dk/Desktop/ngs/bfast-0.6.3c/Makefile
83,87c83,87
< ACLOCAL = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run aclocal-1.10
< AMTAR = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run tar
< AUTOCONF = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run autoconf
< AUTOHEADER = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run autoheader
< AUTOMAKE = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run automake-1.10
---
> ACLOCAL = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run aclocal-1.10
> AMTAR = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run tar
> AUTOCONF = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run autoconf
> AUTOHEADER = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run autoheader
> AUTOMAKE = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run automake-1.10
91c91
< CFLAGS = -Wall -g -O2 -pthread
---
> CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64
110c110
< LIBS = -lz -lm -lbz2
---
> LIBS = -lbz2 -lz -lz -lm
112c112
< MAKEINFO = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run makeinfo
---
> MAKEINFO = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run makeinfo
118c118
< PACKAGE_STRING = bfast 0.6.4e
---
> PACKAGE_STRING = bfast 0.6.3c
120c120
< PACKAGE_VERSION = 0.6.4e
---
> PACKAGE_VERSION = 0.6.3c
125,129c125,129
< VERSION = 0.6.4e
< abs_builddir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_top_builddir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_top_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.4e
---
> VERSION = 0.6.3c
> abs_builddir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_top_builddir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_top_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.3c
152c152
< install_sh = $(SHELL) /Users/dk/Desktop/ngs/bfast-0.6.4e/config/install-sh
---
> install_sh = $(SHELL) /Users/dk/Desktop/ngs/bfast-0.6.3c/config/install-sh
173a174
> scripts/solid2fastq.pl \

[nilshomer]

Try setting the CFLAGS back to "CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64" and see what happens.

[davidkip]

OK, I've tried that now.

Fresh unpack of each tarball, ./configure in each, then edit the Makefile for the new version to have the addition -m64 etc tag.

No success, but I think I see why from the compilation output. It looks like the flag isn't passed to gcc just by altering this Makefile [I am a complete newbie w.r.t. C compilation BTW]. So the output for the older (working) version is:


$ make
make all-recursive
Making all in bfast
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64 -MT AlignedEnd.o -MD -MP -MF .deps/AlignedEnd.Tpo -c -o AlignedEnd.o AlignedEnd.c
mv -f .deps/AlignedEnd.Tpo .deps/AlignedEnd.Po
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64 -MT AlignedEntry.o -MD -MP -MF .deps/AlignedEntry.Tpo -c -o AlignedEntry.o AlignedEntry.c
...etc.

The output for the new version after editing the Makefile is

$ make
make all-recursive
Making all in bfast
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -MT AlignedEnd.o -MD -MP -MF .deps/AlignedEnd.Tpo -c -o AlignedEnd.o AlignedEnd.c
mv -f .deps/AlignedEnd.Tpo .deps/AlignedEnd.Po
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -MT AlignedEntry.o -MD -MP -MF .deps/AlignedEntry.Tpo -c -o AlignedEntry.o AlignedEntry.c
...etc.

Does that help?

[nilshomer]

In configure.ac, set "CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64". The run "sh autogen && ./configure".

[davidkip]

Success!

In the config.ac file for older versions there is this line:


extended_CFLAGS="-m64 -D_FILE_OFFSET_BITS=64";

but in the latest version it is commented out to this:

extended_CFLAGS="";# "-m64 -D_FILE_OFFSET_BITS=64";


Changing it to the earlier version of this line results in the m64 flags being passed to gcc, and the eventual build of the current release now works perfectly. Sorted!

Thanks for your help Nils.

[seeker]

I am having a similar error running bfast match. It will go through one index (of 10 indexes) fine and then give the same malloc error followed by a message saying that it could not allocate enough memory on the second index.

I edited the config.ac to:

extended_CFLAGS="-m64 -D_FILE_OFFSET_BITS=64";

which got me through the first couple indexes and then the same error. I then edited:

CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64

Which got me through the first four but then gave an error. Interestingly, if I leave out -i 1-10 (or whatever) it goes through the full list of indexes fine.


ps - sorry if there are any misspelled commands, my workstation running the analysis isn't currently connected to the web, so I'm copying by eye!

[seeker]

actually, it failed again with my real data..(above was with a small test file). any suggestions here would be great.

[nilshomer]

actually, it failed again with my real data..(above was with a small test file). any suggestions here would be great.

I would need a way to reproduce:

https://sourceforge.net/apps/mediawiki/bfast/index.php?title=FAQ#Please_read_this_before_submitting_a_help_request

I would be happy to help in that case.

[seeker]

It looks like I was doing something stupid by telling it to use more threads than made sense. Reducing that to the number of cores on my processor fixed the problem.