BEAUti memory hungry

[rbouckaert]

Ralf Mueller reports on the user list:

"This is what I did:

• imported 1522 nexus files (8 species each and an average length of 3500 nt -> This took about 2 hours.)

• renamed the clock model of the first partition, marked all partitions and clicked "Link Clock Models".

• after about another 3 hours of loads of output on the console, 100% CPU usage and up to 70 GB memory usage, CPU usage dropped to 0 again and the last message I get is:

  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AWT-EventQueue-0" " So, there are about 8x3500x1522 =42 616 000 characters, which should take for partials (since it uses patters not all sites) no more than 8 (double) x 2 (partials+stored partials) x 4 (DNA) x 7 (internal nodes) x 3500x1522 = 2 386 496 000 bytes ~ 2.3GB. For tip states, much less is required but certainly not more. The Alignment may take up that amount of space again, so it should fit inside 3*2.3GB = 6.9GB, which is much less than the available 70 GB. We should investigate what uses that much memory.

[rameshbalan]

Very Similar issue.

• STARBeast2 template.
• 1364 fasta files (5 species each x ~ [200 -1200] amino acids) imported as individual partitions. ~ 3hours.
• linked tree models together ~4hours.

Error:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "AWT-EventQueue-0"

System Specs: 128GB RAM 24 cores

Is there a workaround to solve this issue? or is there an other way to prepare xml file with starbeast2 template? Please let me know if you need more information.

[rbouckaert]

Something you can try is to provide more memory to BEAUti. Since java is used, it requires to be specified how much memory is maximally available. This can be done by editing the script used to start up BEAUti, which differs a bit per operating system:

• On Windows, BEAST\bat\beauti.bat replace java -cp ... with java -Xmx32g -cp ... in two places, to reserve up to 32GB to BEAUti. Start BEAUti through the batch file, not the BEAUti.exe file.

• On OS X: /Applications/BEAST 2.6.3/bin/beauti

• On Linux: beast/bin/beauti the last few lines says -Xmx8g (indicating 8GB of memory is available) and you can edit it allow more memory. Start BEAUti through the script (not the icon on OS X).

Hope this helps for now.

[rameshbalan]

Thanks for solution! So, I made a copy of the beauti on my linux machine and named it beauti64. I changed -Xmx8g to -Xmx64g on this file and started the program from the terminal.

• Imported the alignment with StarBEAST2 template. ~ 1hour
• Linked Clock Models together. ~ 3 hours.
• Linked Tree models. I still got the OutOfMemory Error.

I noticed that beauti64 is still not actually using 64GB. htop shows only ~16GB of RAM usage.

BEAST v2.6.2 StarBEAST2 v0.15.5

Is there anything else that I can try to overcome this error? Thank you!

[rbouckaert]

Did you restart BEAUti between setting up the different analyses?

A quick and dirty fix specifically for setting up a linked tree analysis: you could edit the BEAUti templates in beast/templates and replace all instance of .t:$(n) with .t:tree. This phrase is used by BEAUti to distinguish different tree partitions, and by fixing it to the tree partition it does not distinguish tree partitions so trees will be linked.

[rameshbalan]

Did you restart BEAUti between setting up the different analyses?

Yes, I closed all instances of beauti and started the modified beauti file with 64GB from the terminal

I made a copy of the StarBEAST template and replaced all the instances of .t:$(n) with .t:tree. I started beauti64 from the terminal, loaded the modified template and imported all the alignments. Surprisingly, the import was complete in less than 10 mins. Side Note: It still takes ~3hours to link the clock models.

However, when I click Site Models tab to modify parameters, it takes about ~30 mins to switch from Partitions tab to Site Models tab and throws a OutOfMemoryError. Still no luck creating the XML file. I am happy to try any other suggestions. Thank you for your constant support.

[rbouckaert]

A short cut to link clock models would be to replace .c:$(n) with c:clock in all templates. That would not fix the site model memory issue though. If there is a specific model you are after for all partitions, you could set this up in the standard template (beast/templates/Standard.xml), and remove the substitution model template (beast/templates/SubstitutionModel.xml). If you are unsure about the XML, just set it up for single partition, say partition xyz first, then save as XML, and copy the part for the site model in Standard.xml, then replace .s:xyz with .s:$(n). Hope this helps.

[rameshbalan]

I was a little confused about the templates. I was working with the incorrect template and got errors that I didn't understand.

However, I got the right Standard template from StarBEAST2, modified clock and tree to .c:clock and .t:tree and gammaCategoryCount to 4 and saved it. I used this template and was able to create the xml file.

Note: I checked the sitemodel tab with a smaller dataset and made sure the template works. But I didn't load the sitemodel tab with the ~1300 dataset and skipped straight to clock and other tabs.

Thanks for helping me out once again!