ProbABEL memory allocation malloc mmap

Questions about ProbABEL are welcome here.
Forum rules
Please remember not to post any sensitive data on this public forum.
The first few posts of newly registered users will be moderated in order to filter out any spammers.

When get a solution to the problem you posted, please change the topic name (e.g. from "how to ..." to "[SOLVED] how to ..."). This will make it easier for the community to follow the posts yet to be attended.
manondelahaye
Posts: 3
Joined: Tue Feb 15, 2011 8:09 am

ProbABEL memory allocation malloc mmap

Postby manondelahaye » Tue Feb 15, 2011 8:22 am

Hi,
I am a french PhD student working at the International Agency for Research on Cancer. We are currently using probabel for association analysis.
I run imputation using mach.

I could run ProbABEL easily for chromosomes 19,20,21,22 but got into trouble for chromosomes 18 to 1. We are running on both a MacOSX machine and a linux cluster, both 64bits.
We compiled the source code with the –m64 flag to get a 64 bits version.

However when running chromosome 18 (for example) I get an error like
palogist(827) malloc: *** mmap(size=18446744066193199104) failed
(error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mematrix(nr,nc): cannot allocate memory (6926,348813)


Do you have any idea where that can come from ? Thanks a lot for your help.
Regards,
DELAHAYE Manon

P.S
Some additional information

For chromosome 18 (smallest one of the ones that do not work), we have 348813 SNPs and 7006 individuals.
probABEL version :
- ProbABEL v. 0.1-9c (September 01, 2010) on my machine
- ProbABEL v. 0.1-3 (Nov 11, 2009) on the cluster
RAM available : up to 48Go on the cluster, 12Go on my MacOSX machine.

Thank you very much.

yurii
GenABEL developer
GenABEL developer
Posts: 263
Joined: Fri Jan 21, 2011 5:20 pm

Re: ProbABEL memory allocation malloc mmap

Postby yurii » Wed Feb 16, 2011 10:34 pm

what is the exact command you run ProbABEL with?

do you use DatABEL format of the data (conversion achieved by mach2databel command of GenABEL) or plain MACH? The suggestion is to try DatABEL format, as this is the one which is faster and very economical on RAM

Yurii
Note that (Gen)ABELs are dynamically developing; while this post is intended to provide full information at the time of posting, please read on further posts, if any, as the topic may be updated with novel solutions at a later stage.

best regards,
Yurii

manondelahaye
Posts: 3
Joined: Tue Feb 15, 2011 8:09 am

Re: ProbABEL memory allocation malloc mmap

Postby manondelahaye » Mon Feb 28, 2011 7:57 am

Hi,
Sorry for the late answer.
Our command was using palogist with all the required parameters, no optional ones.
I don't use DatABEL, I run directly from the info and the dosage file coming out from MACH.

As we did not have a lot of time to work this around, though I still think it is something with the 32/64bits version, we decided to parallelize the process.

In other words, for the chromosomes that were giving this kind of error (memory allocation), we split the file into different subfiles and run many times probabel. It worked fine that way but it is more work to achieve... Also that made me think that it is probably not a problem with the format of the file or the covariates. If it was, it would not be working with the subfiles.

Do you have any idea where that can come from ?

One other thing I wanted to add, it would be a lot easier if palogist could accept .gz files as they can come like that out of MACH. It would save a lot of memory storage.

Thanks for your help.
Manon

yurii
GenABEL developer
GenABEL developer
Posts: 263
Joined: Fri Jan 21, 2011 5:20 pm

Re: ProbABEL memory allocation malloc mmap

Postby yurii » Thu Mar 24, 2011 7:50 pm

I still think it is something with the 32/64bits version


If you have a 32-bit version, then you can not have files >4Gb. If you do get files right, and get memory allocation message, this means that you are low on RAM. In this case, a solution would be to migrate your MACH/IMPUTE files to DatABEL's 'filevector' format -- see help(mach2databel) of GenABEL. Using filevector format, ProbABEL can operate in very low RAM mode -- using less than 512 Mb RAM. Be careful -- as far as I remember, there is a problem with running Cox using filevector files.

One other thing I wanted to add, it would be a lot easier if palogist could accept .gz


I agree -- this would be terrific and will save a lot of HDD space. Even for filevector binary format, gzipping allows about x5 compression!
Note that (Gen)ABELs are dynamically developing; while this post is intended to provide full information at the time of posting, please read on further posts, if any, as the topic may be updated with novel solutions at a later stage.

best regards,
Yurii

manondelahaye
Posts: 3
Joined: Tue Feb 15, 2011 8:09 am

Re: ProbABEL memory allocation malloc mmap

Postby manondelahaye » Mon Mar 28, 2011 8:03 am

Hey
Thanks for the answer. The thing is I run palogist on a cluster with machines that have up to 48Go RAM and are 64bits. So files > 4Gb should not be a problem.
This is why I don't get my error.

Manon

lckarssen
Site Admin
Site Admin
Posts: 322
Joined: Tue Jan 04, 2011 3:04 pm
Location: Utrecht, The Netherlands

Re: ProbABEL memory allocation malloc mmap

Postby lckarssen » Mon Mar 28, 2011 8:46 pm

Can you give some more specifics on your linux cluster?
    - Which kernel are the compute nodes running (output of uname -a)?
    - Which linux distribution?
    - What is the output of file palogist?
Other ideas:
    - Upgrade ProbABEL to v 0.1-9d (the latest) on you cluster. Take care, at present we have a bug where probABEL is missing two files, see this topic.
    - Since you specify the -m64 flag, does that mean you're cross-compiling on a 32 bit machine? For our own 64bit binaries (compiled on the same 64 bit machine as where ProbABEL runs) I've never used that flag.

All in all It is strange that some chromosomes seem to run just fine...
-------
Lennart Karssen
PolyOmica
The Netherlands
-------


Return to “ProbABEL”

Who is online

Users browsing this forum: No registered users and 1 guest