keymapperorg
Feature request: New action(s) - click/tap items in other apps
Developer TODO
- [x] Create UI element repository. Store it in a database so it isn't always loaded into memory if the user isn't on the screen element page. Also allows elaborate filtering and in case the app crashes you don't lose the recorded elements.
- [x] Create screen to record UI elements. Set a timer of 2 mins.
- [x] Create Interact with screen element page.
- [x] Create action in Choose Actions. ~~This action should store the path all the way up to the root so it can be found efficiently.~~ This is premature optimization.
- [x] Finish TODOs in the code
- [x] Add to changelog and website.
- [x] Title at top of app screen is wrong
- [x] Max 1 line on description
- [x] Action/constraint shortcuts max 2 lines
- [x] Automate detects the element you interact with when recording properly.
- [x] Add tooltip and hint text
- [x] Auto suggest the name
- [x] Be more lenient with detecting and not all fields need to match.
- [x] Adapt the layout for horizontal.
- [x] Show additional elements by default if no elements recorded.
Great app. I use a closed source app (Automate) for custom keyboard shortcuts, but if KeyMapper implements this feature I can replace Automate:
Action(s) that allows for clicking/tapping items in other apps. There can be a button to record the interaction in the other app so KeyMapper knows what to click. This will be good for apps that use Android provided buttons, switches, and menus. To implement this, I'm guessing that while KeyMapper is running in the background it will need a way to keep track of the current app activity so that custom shortcuts wont bleed between apps.
Another idea would be if KeyMapper fails to detect any clicking actions while recording the interaction, it could fall back on clicking a user designated X,Y coordinate on the screen. This might be useful for apps such as games that don't use Android provided buttons, switches, and drop menus.
actiontype = SCREEN_INTERACTION
performAction on accessibility node. Let them tap on a view in an app and Key Mapper will detect this. throw action error if isn't in the correct app.
- [ ] Have textinputlayouts with EditTexts and Switches inside to pick..
These are all compulsory...
- [ ] action (click, long click, scroll forward, scroll backward, focus, select, dismiss)
- [ ] package name
- [ ] UI element text optional...
- [ ] UI element id (viewIdResourceName). Not all views have this and only show this option on API 18+.
- [ ] UI element hinttext
- [ ] UI element tooltipText
Thank you for looking into this feature request, as well as the tap coordinates action (it's very cool). How feasible would an action that clicks a user provided/cropped .png file of an UI element be? E.g. the user gives Key Mapper a cropped screenshot of a button to tap, Key Mapper searches for it on the screen (assuming it has the accessibility service), and if found Key Mapper taps the center of it?
It is technically possible (not sure on the latest Android versions) but there is no elegant way to do it. Lets say the screen is just a picture, finding another picture of random resolution and accuracy in another picture isn't trivial. What is the use case for providing a cropped .png of a UI element? If the UI element is a standard UI element that subclasses View.java then the accessibility can find it with the accessibility nodes API thing.
Sorry I was unclear in my last reply, I thought "UI element" was a generic term rather than specific to Android. Some usecases I can think of that don't involve Android UI elements are tapping a button on a webpage (e.g. GH notifications) or a VNC client connected to a remote host. A .png search feature could be used as a fallback when a normal Android UI isn't being used
Yeah, UI element is a generic term. I'm not sure which terminology to use so I'll just use "UI element". The user will have to be a bit familiar with how accessibility service nodes work to understand how this feature works. It is up to the app developers to make their custom Views/UI elements behave nicely with accessibility services, which is why UI elements in some apps can't be interacted with by an accessibility service. The standard views already behave nicely like Buttons and RecyclerViews. Good point, there are useful use-cases for this but I am very unlikely to pursue it since it isn't trivial and there isn't much demand for it.
I thought I'd throw the idea out there. Thank you for the work you put into this great app.