BEAST process killed on logout (LINUX)

[tgvaughan]

Command-line BEAST processes are being killed on logout on my Debian machine, even when launched from the command line using nohup or from within a screen/tmux session.

To replicate:

1. open an xterm and cd to ~/.beast/2.4/BEAST/examples,
2. run an analysis using nohup, e.g. nohup beast testCoalescent.xml &,
3. logout and log back in,
4. observe that the beast process did not survive logout.

The workaround is to explicitly prevent beast from communicating with the X server, either by using DISPLAY="" beast testCoalescent.xml or java -Djava.awt.headless=true -jar ~/.beast/2.4/BEAST/lib/beast.jar testCoalescent.xml.

This may not actually be a beast issue, as systemd has recently made some changes to which processes it kills by default on logout. On the other hand, it seems as though the problem is due to some slight interaction that BEAST has with the X server even when no GUI components are shown, so it may be possible to fix this by identifying and moving the (hypothesized) calls so that they are never called when no GUI is required. (I've had a brief look and can't find anything though.)

[tgvaughan]

I should also add that this is new behaviour. I've used this same approach (sans fiddling with DISPLAY or java.awt.headless) to running beast jobs for years without problem.

[rbouckaert]

Anecdotal evidence that this is a systemd/Linux issue is that I used to be able to ssh into a Linux machine with X enabled that killed any BEAST job started in the background on logout. If I ssh with X not enabled, jobs remains alive when logging out -- no display settings required.

[alexeid]

I don't think Java programs without any AWT calls will have this problem (easily tested). So we are probably inadvertently calling some Swing or AWT code even when not popping up windows.

Alexei

On 8/11/2016, at 6:17 PM, Remco Bouckaert [email protected] wrote:

Anecdotal evidence that this is a systemd/Linux issue is that I used to be able to ssh into a Linux machine with X enabled that killed any BEAST job started in the background on logout. If I ssh with X not enabled, jobs remains alive when logging out -- no display settings required.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[tgvaughan]

@alexeid I tested whether a looping hello world Java program had the same issue yesterday when I was trying to find a solution. The program wasn't killed on logout.

[walterxie]

Hi, Tim,

If I understand the problem correctly, don't you need to use either "screen" or other methods to make it run on the background? Otherwise, you have to install a job scheduling system to create the cluster.

Walter

From: Tim Vaughan <[email protected]mailto:[email protected]> Reply-To: CompEvol/beast2 <[email protected]mailto:[email protected]> Date: Wednesday, 9 November 2016 11:53 am To: CompEvol/beast2 <[email protected]mailto:[email protected]> Subject: Re: [CompEvol/beast2] BEAST process killed on logout (LINUX) (#635)

@alexeidhttps://github.com/alexeid I tested whether a looping hello world Java program had the same issue yesterday when I was trying to find a solution. The program wasn't killed on logout.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/CompEvol/beast2/issues/635#issuecomment-259285155, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFehpsG-fBp9p-_b0VZMwlDmmAPnpIb1ks5q8P1egaJpZM4Kr81k.

[rbouckaert]

Possibly, Class.forname calls in AddOnManager are the culprit. These calls are equivalent to Class.forName(className, true, currentLoader) where the 'true' means the class will be initialised. Replacing them by Class.forName(className, false, Reflection.getCallerClass()) may fix this? BTW if a package creates an AWT object, this will be hard to prevent.

[tgvaughan]

One (incredibly heavy-handed) approach to solving potential problems caused by third party packages would be to explicitly set the headless property to true iff BEAST determines that it doesn't need to display an opening dialog, file chooser or console dialog. As well as solving the problem, this would prevent packages run via BEAST's Runnable system from creating dialogs. (I can't off the top of my head think of why you might want to do this, but this is probably just lack of imagination on my part.)

[tgvaughan]

Ok, this one's too subtle for me. I tried to identify the exact problem using a bisection search but the only conclusions I've come to are that:

1. This has nothing to do with the startup dialogs.
2. If java.awt.headless is set to true AFTER XMLParser::parseFile is called from BeastMCMC, the problem persists.
3. If java.awt.headless is set to true BEFORE XMLParser::parseFile is called, the problem goes away.

Actually, I went to a finer level and found that the problem goes away as long as headless=true is set before the call to XMLParser::processPlates in XMLParser::parseFile, but commenting out the call to processPlates does not fix the problem. This suggests to me that processPlates is not the only culprit - merely the one that is called first.

[tgvaughan]

@rbouckaert I don't think AddOnManager is the problem either. Turning on headless mode after the packages were loaded still fixed the issue.

[dasice]

Hey I'm just some random guy from the Internet but I'm having a similar problem with building a Flash gui. In particular I'm starting a java vm running ant which calls the mxmlc Flash compiler. I have the same problem though - when I use tmux and log out the mxmlc compiler dies because this magic java.awt.headless variable somehow is not set correctly.

Funny enough when I call the compile with nohup or use gnu screen everything works fine.

I actually fixed it by supplying the argument -Djava.awt.headless=true to the ANT_OPTS java VM argument.

There's something a little weird about how tmux interacts with java when it tries to call X. Maybe tmux tries to be an X server if the DISPLAY is set? Donno, good luck.

[SimonGreenhill]

I had a similar issue with this, turns out the command being used was ssh -X (=Enables X11 forwarding). Removing the -X (or replacing with lowercase -x) solved it.

[tgvaughan]

Thanks for the heads up @dasice, but our problem isn't specific to tmux: nohup and screen both produced the same problem. @SimonGreenhill: this is definitely the same problem. If DISPLAY points to an X server, either because you're on a desktop with X running, or because you're using ssh with X forwarding from such a desktop, BEAST tries to be non-headless. Prepending the launch command with DISPLAY="" should always fix the problem.