libretro
Fix mouse grab behavior on Android
Description
In #14016, @schellingb posted a branch (https://github.com/schellingb/RetroArch/tree/android_grab_mouse) that implemented a grab_mouse interface for Android. I forked and tested it extensively on an Android 10 device, and ended up refactoring the RetroActivityFuture a bit so that it worked as expected.
I ran into a few things: looking at the ADB logs, I noticed that, for some reason, enabling "Immersive Mode" and attempting to call requestPointerCapture on the window's decor view was inconsistently working. One of the only references I could find relating to this behavior was this ImmersiveActivity example from the Android source code, where they used a Handler to toggle the UI changes with a 300ms delay. I ended up doing the same thing in the RetroActivityFuture, using a Handler to toggle UI changes (including mouse capture and "immersion mode") with a 300ms delay, and it seems to work pretty well.
I also made the input_auto_mouse_grab config setting default to true, for Android only, based on the conversation in that issue. That way, mouse capture works by default, but you can still disable the setting if (for some reason) you don't want the mouse to be captured by RetroArch.
I tested this using the Mame 2003-Plus and FinalBurn Neo cores, directly through the RetroArch APK, and indirectly through the Daijishou front-end, which uses am start to launch the RetroActivityFuture activity. Everything works as expected!
~Controlling the menu UI with a mouse in Android still doesn't seem to work, however - someone with more experience working in the RetroArch codebase might have an easier time fixing that.~ Fixed in e60f04c
Related Issues
#14016
Reviewers
@schellingb - thank you very much for co-authoring this in your android_grab_mouse branch, and for implementing grab_mouse, otherwise I wouldn't have known where to start!
Hi there.
Thanks for pushing this forward! I kinda lost interest after seeing the remaining issues and the overall inconsistent behavior on Android with a mouse attached versus a desktop PC running RetroArch in fullscreen.
I also made the input_auto_mouse_grab config setting default to true, for Android only, based on the conversation in that issue. That way, mouse capture works by default, but you can still disable the setting if (for some reason) you don't want the mouse to be captured by RetroArch.
Is this a solution or a workaround? I was hoping for no need for this as it isn't needed on desktop platforms. Still, this is probably better than having to explain users to need to toggle the setting if they want to use a mouse. Although existing RetroArch installations won't get the setting switched automatically.
Controlling the menu UI with a mouse in Android still doesn't seem to work, however - someone with more experience working in the RetroArch codebase might have an easier time fixing that.
Last time I noticed the menu does react to mouse movement and clicks but it seems utterly confused about the mouse position or its movement. For me it just jerked the menu around when I tried it back then, is this still the same?
Does anyone know how other (non-desktop) platforms handle a mouse? I'm interested in how it runs in something like DRM/KMS, which is Linux without a desktop manager. For example, does Lakka on a RaspberryPi support a USB mouse to control the menu and games?
Edit: Also wanted to add thanks to @worleydl who initially took a stab at implementing the interface, otherwise I wouldn't have known where to start :-)
@schellingb I think this commit (e218a964612478a4bc02ce0fb2ab28b0844c0f60) might have the behavior you're looking for, I'm gonna test it and let you know
@schellingb controlling the menu with a mouse works now, in this commit: cd47e27 🎉
The issue was that android_input_state wasn't handling RARCH_DEVICE_MOUSE_SCREEN.
I also tested the commit to always capture the mouse on focus, regardless of the input_auto_mouse_grab config setting (https://github.com/libretro/RetroArch/commit/e218a964612478a4bc02ce0fb2ab28b0844c0f60). It works, but I'm on the fence about having that setting essentially be a no-op for all non-windowed platforms. I'm worried about someone (for some reason) wanting to keep it disabled, and then this change would prevent that...
We could remove the config option entirely for non-windowed platforms as a part of this PR, but it feels like there's a lot going on here already. What do you think?
This is a long shot but is there anything specific that made the initial report Android-only aside from the device it was tested on? I believe the same issue exists on iOS and wondering if it could be ported: https://github.com/libretro/RetroArch/issues/15483 (I'm also willing to test).
@TyroneSlothrop unfortunately everything here is Android-specific (input drivers, frontend drivers, etc), so it can't be ported to iOS. Looking at the input drivers for Cocoa, it seems like iOS is set up for mouse support... I'll ask some questions in that issue you posted
thank you for working on this. just tested it and i have some feedback
- the mouse sensitivity is way too low (on the menu atleast), even turning the android mouse speed setting to the max didn't help
- touch input is still broken if you turn on "mouse support" on the menu (though it was already broken previously anyway)
https://github.com/libretro/RetroArch/assets/43626415/ca4c1ec0-a2b0-4f81-9e05-3d5934c0384a
@LuanVSO thank you for the video - I'm trying to see if it's something I can do to speed up the cursor, but I haven't had any luck yet...
@schellingb I ended up reverting that "always capture the mouse regardless of the input_auto_mouse_grab config value" commit ( https://github.com/libretro/RetroArch/commit/e218a964612478a4bc02ce0fb2ab28b0844c0f60), the way it is setup now makes it easy when debugging mouse behavior like https://github.com/libretro/RetroArch/pull/16203#issuecomment-1933118104
@LuanVSO I think I figured out the pointer speed issue, can you give the build on this commit a try? https://github.com/libretro/RetroArch/pull/16203/commits/a4bb625d35fd372e3a808b7eb15dae29ffde06d1
@LuanVSO I think I figured out the pointer speed issue, can you give the build on this commit a try? https://github.com/libretro/RetroArch/pull/16203/commits/a4bb625d35fd372e3a808b7eb15dae29ffde06d1
- Mouse speed is fixed now
- touch behavior is better as well
- even navigating the menu on samsung dex works great now
But
- when the mouse is captured it can't go past about 60% of the screen horizontally
- it appears that the position of the last touch event gets "saved" and the position the the cursor keeps fliping between the mouse position and the last touch event (it keeps clicking stuff too)
https://github.com/libretro/RetroArch/assets/43626415/3e3b332d-6ea3-4e43-b7f3-0e60ee66009c
@LuanVSO this build uses the actual width/height from the video resolution, give it a shot: https://github.com/libretro/RetroArch/pull/16203/commits/9a1c30ab373008de3c5ace9a7727760e9a754ca8
I haven't looked into any of the touch behavior yet, I'll take a look at some point
The mouse capture is good, the remaining issues are marginally related but weren't introduced by this pr. I think it's fine to merge it.
@LibretroAdmin I’m good with merging it in it’s current state, I know @schellingb didn’t want the final result to depend on having the input_auto_mouse_grab config value, but I think we could try and make that change downstream from this PR. I think having it default to true for Android (going forward) is good enough for now, and people can always enable it themselves if they already have a config
I would love to have this so i can finally start enjoying some point and click adventures :)
@LibretroAdmin I have been using this branch to play point-and-click DOS games and trackball games on Android for a few weeks now, and it works really well. I think it's good to be merged
Honestly mouse should be captured by default in all platforms. It's the cause of a lot of issues in desktops too when using cores hiding the mouse in windowed RA or fake Fullscreen (vast majority of desktops these days) iirc. There are worse problems (saw a lot of funny behaviors in KMS\DRM Linux because it bypasses the hardware quirks library libinput usually loaded by gnome\window manager), but even if everything else is working, not having it on can cause bugs in those cases... as this pr showed for Android and others showed for Wayland (mouse capture was a no op in Wayland until a recent series of fix commits, so that 'just' allowed mouse capture to do its job IF turned on).
