jmonkeyengine Mouse pointer moves to centre of canvas on click in OpenGL3 with Swing

With JME3.7 we can now embed OpenGL3 canvas in a Swing window. I have noticed an issue with the mouse with this configuration.

Steps to reproduce:

Use JME 3.7.0-stable
Be running OpenGL3
Have a jme canvas embedded in a swing window
Listen for jme mouse events using InputManager.addListener(ActionListener)
Left click on the canvas
In the ActionListener, get the position of the mouse using InputManager.getCursorPosition()

Observed The mouseDown (i.e. keyPressed=true) will have the correct mouse position. By the time the mouseUp event is processed (i.e. keyPressed=false), the position of the mouse has been moved to the centre of the canvas.

The mouse is physically displayed in the centre of the canvas
InputManager.getCursorPosition() will return a mouse position in the centre of the canvas

Notes:

This does not happen if you are not using Swing
This does not happen in OpenGL2 with Swing
Seen on both Windows 10 and current Debian linux stable (12.7)

A workaround (more a hack) is to

Store the location of the mousedown event
In the mouseup event, use java.awt.Robot to move the mouse back to where the mousedown event happened.

Nov 12 '24 11:11 terranprog

I should have added that I am using FlyByCamera with dragToRotate set to true. This turns out to be relevant.

A few things I have learnt so far:

The reason for the difference in behaviour between jme3-lwjgl and jme3-lwjgl3 is that they use different instances of com.jme3.input.MouseInput

jme3-lwjgl uses an instance of com.jme3.input.lwjgl.LwjglMouseInput (even if running in a Swing window) jme3-lwjgl3 uses an instance of com.jme3.input.awt.AwtMouseInput

The unwanted recentering of the mouse cursor that I am seeing happens in com.jme3.input.awt.AwtMouseInput.setCursorVisible.

component.setCursor(newVisible ? null : getTransparentCursor());
    if (!newVisible) {
        recenterMouse(component);
    }

So every time the mouse cursor is set to not visible the cursor is moved to the centre of the canvas.

com.jme3.input.lwjgl.LwjglMouseInput has different code in setCursorVisible: Mouse.setGrabbed(!visible);

com.jme3.input.awt.AwtMouseInput.recenterMouse has this comment: // MHenze (cylab) Fix Issue 35: A method to generate recenter the mouse to allow the InputSystem to "grab" the mouse

According to git this Fix has been there since at least 2011.

So in summary, the two instances of MouseInput have different ways of "grabbing" the mouse.

Secondly... The FlyByCamera has code to hide the mouse cursor when dragToRotate is initiated.

FlyByCamera.onAction(String name, boolean value, float tpf) {
    if (name.equals(CameraInput.FLYCAM_ROTATEDRAG) && dragToRotate) {
        inputManager.setCursorVisible(!value);

The interesting thing is that FlyByCamera treats left mouse clicks as ROTATEDRAG events. inputManager.addMapping(CameraInput.FLYCAM_ROTATEDRAG, new MouseButtonTrigger(MouseInput.BUTTON_LEFT));

This seems questionable. A mouse click is not a mouse drag.

I can see at least two possible fixes for this issue:

Improve FlyByCamera so it can differentiate between mouse clicks and mouse drags. That way it would only hide the mouse cursor when the user actually clicks and drags. Which I assume is the intended behaviour.
Change AwtMouseInput so it doesn't always centre the mouse when hiding the mouse cursor. Perhaps this behaviour is only required if the mouse is outside the canvas? Or perhaps it could move the mouse to the current mouse position instead of the centre of the canvas? It is also possible the original fix is no longer required.

Nov 12 '24 21:11 terranprog

Hi, here is a test class to replicate the issue.

public class Test_SafeCanvas extends SimpleApplication {

    public static void main(String[] args) {
        AppSettings settings = new AppSettings(true);
        settings.setGammaCorrection(false); /* for lwjgl3 */
        settings.setResolution(640, 480);

        final Test_SafeCanvas app = new Test_SafeCanvas();
        app.setPauseOnLostFocus(false);
        app.setSettings(settings);
        app.createCanvas();
        app.startCanvas(true);

        JmeCanvasContext context = (JmeCanvasContext) app.getContext();
        Canvas canvas = context.getCanvas();
        canvas.setPreferredSize(new Dimension(settings.getWidth(), settings.getHeight()));

        try {
            Thread.sleep(3000);
        } catch (InterruptedException ex) {
            ex.printStackTrace();
        }

        JFrame frame = new JFrame("Test");
        frame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
        frame.addWindowListener(new WindowAdapter() {
            @Override
            public void windowClosing(WindowEvent e) {
                app.stop();
            }
        });

        frame.getContentPane().add(canvas);
        frame.pack();
        frame.setLocationRelativeTo(null);
        frame.setVisible(true);
    }

    @Override
    public void simpleInitApp() {
        flyCam.setDragToRotate(true);
        flyCam.setMoveSpeed(10f);
        viewPort.setBackgroundColor(ColorRGBA.DarkGray);

        Box b = new Box(1, 1, 1);
        Geometry geom = new Geometry("Box", b);
        Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
        mat.setTexture("ColorMap", assetManager.loadTexture("Interface/Logo/Monkey.jpg"));
        geom.setMaterial(mat);
        rootNode.attachChild(geom);
    }

}

Nov 14 '24 17:11 capdevon

I fixed the issue (for me) by modifying FlyByCamera to only hide the cursor when the view is actually rotated. A pull request is here: https://github.com/jMonkeyEngine/jmonkeyengine/pull/2328

I first tried to fix it by modifying com.jme3.input.awt.AwtMouseInput but couldn't make it work. Also the change required seems more intrusive and more difficult to reason about,

Nov 15 '24 06:11 terranprog

Hello everyone.

It seems that this problem goes beyond simple mouse cursor positioning; It seems that if the flying camera is active this will eventually break the analog inputs.

By adding the following code block to the example provided by @capdevon, you can see that the mouse position is always the same (There is a small change in position, but it is minuscule - AnalogListener):

inputManager.addListener((ActionListener) (String name, boolean isPressed, float tpf) -> {
	Vector2f pos = inputManager.getCursorPosition();
	if (isPressed) {
		System.out.println("[DOWN] :" + pos);
	} else {
		System.out.println("[ UP ] :" + pos + "\n");
	}
}, "Action");
inputManager.addMapping("Action", new MouseButtonTrigger(MouseInput.BUTTON_LEFT));

inputManager.addListener((AnalogListener) (String name, float value, float tpf) -> {
	Vector2f pos = inputManager.getCursorPosition();
	System.out.println("[ANALOG] :" + pos);
}, "Analog");
 inputManager.addMapping("Analog", new MouseButtonTrigger(MouseInput.BUTTON_LEFT));

[ OUTPUT ]

[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
[ANALOG] :(320.0, 240.0)
...

[ NOTE ]

Tests carried out in GNU/Linux, I don't know if this is the same on Windows...

Nov 25 '24 18:11 JNightRider

When the cursor is not visible then there is no cursor position. Asking for the cursor position when the mouse is not visible doesn't make any sense, I think.

My recollection (from 10 years ago or more) is that the cursor position is reset constantly to the center to avoid runout when you hit the edge of the screen and none of your axes update anymore. (Proper way to track mouse when the cursor is invisible is with an analog listener on that axis... but using the value not the cursor position... just like any other joystick axis, etc.)

Dec 23 '24 18:12 pspeed42