Feature: Keyboard shortcuts

[pocmo]

This came up for the reference browser here: https://github.com/mozilla-mobile/reference-browser/issues/575

Since most of those shortcuts would probably invoke UseCases (or indirectly perform things on the SessionManager or Session).. maybe we could just wrap that in a component and with that provide those keyboard shortcuts to all our apps easily.

Related code in Fennec: https://searchfox.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/BrowserApp.java#471-567

https://searchfox.org/mozilla-esr68/rev/13df4aebd54ee9ba73924dd558af289ef515a490/mobile/android/base/java/org/mozilla/gecko/BrowserApp.java#534

---

• [ ] New tab
• [ ] Navigate forward
• [ ] Navigate back
• [ ] Reload
• [ ] Stop loading
• [ ] Private tab
• [ ] Close tab
• [ ] Find in page

┆Issue is synchronized with this Jira Task

[samlh]

Out of curiosity, I was taking a look at how complex this would be to implement, so I thought I'd add some notes on how input handling seems to work:

• Key events get passed to the activity, which in turn forwards it down through the tree of views (standard Android behavior).
• GeckoView swallows (almost) all key events that get passed to it, sending them off to another thread to be processed by the Gecko native code
• The native code implements the standard webpage event handling
• If the event is not handled, it gets passed back out to GeckoInputConnection.performDefaultKeyAction which has a small list of events that get special handling

Call tree: ... -> GeckoView.dispatchKeyEvent -> GeckoView.onKeyDown -> SessionTextInput.onKeyDown -> GeckoEditable.onKeyDown -> GeckoEditable.processKey (mark as handled, and send to gecko thread) -> GeckoEditable.sendKeyEvent -> GeckoEditable.onKeyEvent -> IGeckoEditableChild.onKeyEvent (call into native) -> GeckoEditableSupport::OnKeyEvent ->

• first: TextEventDispatcher::DispatchKeyboardEvent -> TextEventDispatcher::DispatchInputEvent -> ...
• if not handled: IGeckoEditableParent.onDefaultKeyEvent (call into java) -> GeckoInputConnection.onDefaultKeyEvent (send to java ui thread) -> GeckoInputConnection.performDefaultKeyAction

I likely won't have much time to work on this in the short term, but if no one else gets to it before me I can make an attempt at implementing this.

Implementation notes:

• If the shortcuts do not need to be visible to webpages, then it is likely enough to override GeckoEngineView.dispatchKeyEvent (or maybe onKeyDown) so that the event never gets to GeckoView (or maybe override on the wrapping BrowserActivity)
• If there are shortcuts that should be overridable by webpages, then GeckoView needs to be updated to allow further handling in GeckoInputConnection.performDefaultKeyAction
• The handling code should be behind an interface so embedders can do their own thing if they don't like the provided shortcut handling code

[gaul]

@data-sync-user @pocmo This issue is not yet fixed -- is there some reason to close it? It has an open PR asking for directions on how to best complete the feature.

[testman42]

I'm interested in this functionality as well. My use-case is just fenix with generic keyboard connected to tablet via USB hub. @gaul Would you be so kind as to post the link to the PR that you mentioned?

[eUgEntOptIc44]

@data-sync-user @pocmo This issue is not yet fixed -- is there some reason to close it? It has an open PR asking for directions on how to best complete the feature.

Any news on this issue? @data-sync-user please do re-open this. This seems to be still unresolved. Please also see mozilla-mobile/fenix #10217

[eUgEntOptIc44]

@gaul please be so kind to post the link to the PR if you still have it

[eUgEntOptIc44]

@data-sync-user please also see mozilla-mobile/fenix #3729

[cadeyrn]

@eUgEntOptIc44 data-sync-user is a bot. It won't respond…

[gaul]

@testman42 @eUgEntOptIc44 You can use these as starting points:

• https://github.com/gaul/android-components/tree/keyboard/reload
• https://github.com/gaul/fenix/tree/keyboard/basic

However the real work for this issue is defining the interface in android-components that can accommodate many different use cases, e.g., right clicking.

@pocmo Eagerly awaiting your guidance. :smile:

[pocmo]

Re-opening. Sorry for that, @data-sync-user closed some things accidentally when it was introduced.

However the real work for this issue is defining the interface in android-components

That's also how I remember it.

• Android components is very modular. In this architecture a concept-engine implementation exposes a generic listener for hooking into the key presses (which is needed because GeckoView otherwise will always capture them). And then a separate component (something like feature-keyboard) may provide a generic implementation for all engines that support this interface.
• Now the tricky part is that some shortcuts may only work while the user is browsing (e.g. reload) and may not make sense in other screens. While other shortcuts (let's say "open new tab") are expected to work on other screens, like the "home" screen in Firefox too. In addition to that an app may need to hook into those callbacks too, e.g. if you invoke the "open new tab" shortcut then Firefox may want to show the UI for that, which Android Components doesn't know anything about. So this feature may need to provide some callbacks for the app.

However we do not need to tackle all of those at once. We can focus on the concept-engine interface design and then start with only shortcuts that are needed while browsing (e.g. reload, back, forward). And then look at other things later.

The good thing is... if we get this designed and working here then this will work in Firefox Focus too. :)

[ValTM]

I am also very interested in this coming to realization - this would open up Firefox to be the browser on Android tablets, as many of them come with a keyboard now. Yet being unable to use the familiar shortcuts is making the experience very unpleasant.

Looking forward to when this is fixed!

[czlucius]

@testman42 @eUgEntOptIc44 You can use these as starting points:

* https://github.com/gaul/android-components/tree/keyboard/reload

* https://github.com/gaul/fenix/tree/keyboard/basic

However the real work for this issue is defining the interface in android-components that can accommodate many different use cases, e.g., right clicking.

@pocmo Eagerly awaiting your guidance. 😄

Hi @gaul @pocmo I would like to work on this issue. How should I build AC such that my debug version can be integrated into a debug version of Fenix (or reference browser)? From what I know, Fenix pulls AC from Gradle builds, hence I do not know how to test out the change. Thanks

[pocmo]

Hey @czlucius, unfortunately I do not work at Mozilla anymore. However back then what I preferred to do was local releases by doing:

• Changing version here (adding at timestamp or something else that doesn't conflict with actual versions): https://github.com/mozilla-mobile/android-components/blob/main/version.txt
• Running publishToMavenLocal
• Adding mavenLocal() as repository in Fenix (two places) and using my custom version

[csadilek]

Moved to bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1794664

Change performed by the Move to Bugzilla add-on.