nvaccess
NVDA Command Palette
Is your feature request related to a problem? Please describe.
Currently, there are several commands in NVDA that can only be executed if an input gesture is assigned to them. This can present a problem where a user wishes to execute a script that does not have an input gesture assigned, but which they do not wish to give an input gesture, for example because it is rarely used. This may also be problematic for users who have difficulty remembering or executing complex input gestures, for whom there are a limited number of available key combinations to choose from.
Describe the solution you'd like
Introduce a "command palette" that the user can open with a single input gesture (or via the NVDA menu), which presents all scripts that are possible in the current context. Allow the user to type to search for gestures (preferably with some sort of "fuzzy logic" to allow for imprecise matches). Allow the user to activate the selected script, with the effect of the script taking place in the context from which they invoked the command palette. Optionally, display the input gesture(s) associated with the script (according to the method used to open the command palette (desktop layout/laptop layout/touch)) alongside the script.
Describe alternatives you've considered
Add an "Activate" button to the input gestures dialog, which performs the selected script in the context from which the user opened the input gestures dialog.
Additional context
The command palette paradigm is present in many software products. For instance, Visual Studio Code and Windows Terminal both offer such functionality (by default both invoked with Ctrl+Shift+P). Some other screen readers, including Voiceover on the Mac, offer a "Commands Menu" which is functionally equivalent to a command palette.
Example user stories:
- As a user with a cognitive impairment, I want to be able to use natural language to search for NVDA commands so that I can use my computer without struggling to remember a lot of keystrokes.
- As a new NVDA user, I want to be able to search for the command I need so that I can carry out tasks without having to switch to and from NVDA’s documentation.
- As a user with arthritis, I want to be able to execute NVDA scripts without complex keyboard shortcuts that are painful to input.
- As a power user, I want to be able to search for NVDA scripts to help me quickly activate less frequently used actions.
A similar experience, to some extent, is implemented in Command Helper add-on by @javidominguez, present in the Add-on Store.
I think the "Activate" button in the input gesture dialog is also implemented in an add-on, NVDA global commands extension; maybe @paulber19 can confirm.
Hello, indeed, Command Helper implements that feature. I was actually inspired by the VoiceOver palette to develop it. It works through a Virtual Menu, with no visible interface. When the menu is active, a keyboard overlay captures all keystrokes. It also includes voice command search. All of this is intended to ensure that the addon interferes as little as possible with the system state. The problem with displaying a visible dialog with a actívate button is that it attracts the system focus and even if you hide it before launching the execution, the system focus or NVDA focus may not return exactly to the same place it was and that would alter the result of some scripts.
I think that the CommandHelper add-on is very valuable. Also, it would be great if this could be implemented with a visible interface. Years ago, when I created JAWS scripts, a dialog could be created in JAWS to select the command to be run with a visible interface. I think that there are people complaining about NVDA keystrokes, difficult to press or to remember, and hope this issue can be prioritized accordingly. I'm very interested, mostly for new users or even people with low vision, so they can see the available keystrokes easily.
Thinking about this, I think that the suggested alternative of using the input gestures dialog would be nice, to avoid the creation of a different command. Also, some people may be used to this dialog and the search feature is already implemented. If the option is expanded to select a gesture, it may be passed to the script.
The problem may arise with commands without a description, intended to send a gesture, not shown in this dialog.
A predefined description may be used in these cases, like: "Sends x gesture", or these commands may be excluded.
Another problem is caused by commands which require to repeat a gesture, but I think that these commands should have their own scripts.
I've tested in the Python console, and seems that gui.mainFrame.prevFocus can be used to ensure that the focus is set to that object, thought I'm not completely sure if this would work in all cases.
Though generally, when destroying the input gestures dialog, the focus seems to be preserved, though I would need to perform more testing to be sure.
YES, that's correct.
Le 24/09/2024 08:45, Cyrille Bougot a écrit :
A similar experience, to some extent, is implemented in Command Helper https://github.com/javidominguez/commandHelper/ add-on by @javidominguez https://github.com/javidominguez, present in the Add-on Store.
I think the "Activate" button in the input gesture dialog is also implemented in an add-on, NVDA global commands extension; maybe @paulber19 https://github.com/paulber19 can confirm.
— Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/17209#issuecomment-2370329873, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZLFFF3WRY5OKE3DNB57TDZYEC65AVCNFSM6AAAAABOXNJE4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQGMZDSOBXGM. You are receiving this because you were mentioned.Message ID: @.***>
This is a valid use-case and a helpful feature. The Blindpandas team had released an addon for this a while back, I don't think it's compatible with latest versions of NVDA anymore. https://github.com/blindpandas/command_palette
Command Helper use a virtual interface: we can navigate in menus / commands with speech (and maybe braille) feedback, but no dialog is visible at all. On the other side, NVDA global commands extension add-on and Command Palette add-on use real dialogs.
The advantage of a virtual interface with respect to a real dialog is that we have no side effect when calling commands related to the current focus, the navigator object or the review cursor. On the other side, the drawback is that nothing is visible, what may be an issue for visually impaired people or for sighted collaborators.
My personal preference goes for the virtual interface, and I am an occasional user of Command Helper, because I am comfortable enough to work without a visual feedback (even if I have a little vision left.
The problem posed is to activate a script without a input gesture. Today, To find out which scripts can be activated, the user uses the "Input gesture" dialog. It is then that he notices that the script does not have an associated gesture. Two solutions: -, a button in the dialog allows him to activate the script.
- he must use another dialog to activate it.
I think that apart from a technical problem, the choice between the two solutions is obvious.
Le 26/09/2024 13:16, Cyrille Bougot a écrit :
Command Helper use a virtual interface: we can navigate in menus / commands with speech (and maybe braille) feedback, but no dialog is visible at all. On the other side, NVDA global commands extension add-on and Command Palette add-on use real dialogs.
The advantage of a virtual interface with respect to a real dialog is that we have no side effect when calling commands related to the current focus, the navigator object or the review cursor. On the other side, the drawback is that nothing is visible, what may be an issue for visually impaired people or for sighted collaborators.
My personal preference goes for the virtual interface, and I am an occasional user of Command Helper, because I am comfortable enough to work without a visual feedback (even if I have a little vision left.
— Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/17209#issuecomment-2376655053, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZLFFG5R7TIDAQO4G3WTQ3ZYPUHRAVCNFSM6AAAAABOXNJE4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZWGY2TKMBVGM. You are receiving this because you were mentioned.Message ID: @.***>
@paulber19 wrote:
The problem posed is to activate a script without a input gesture. Today, To find out which scripts can be activated, the user uses the "Input gesture" dialog. It is then that he notices that the script does not have an associated gesture. Two solutions: -, a button in the dialog allows him to activate the script. - he must use another dialog to activate it. I think that apart from a technical problem, the choice between the two solutions is obvious.
Even if you are replying to my message, you seem to totally ignore the virtual interface proposal.
Have you tested Command Helper add-on? If yes, why have you preferred the dialog solution in your add-on? You have surely had good reasons, but it would be interesting to make them all explicit here so that they are shared with everyone.
Re the choice between input gesture dialog and another dialog, again explicit is better than implicit. So what seem obvious to me is that if we go forward with a dialog solution, your solution is the best one, i.e. add one more button in the input gesture dialog rather than creating a brand new dialog.
Sorry, but I think I expressed myself incorrectly. Speaking of another dialog, I meant to force the user when he is in the "Input gesture" dialog to use another means to activate a script without having to assign a command gesture , whether this dialogue is visible or invisible. I think this answers all the questions.Le 26/09/2024 18:02, Cyrille Bougot a écrit :
@paulber19 https://github.com/paulber19 wrote:
The problem posed is to activate a script without a input gesture. Today, To find out which scripts can be activated, the user uses the "Input gesture" dialog. It is then that he notices that the script does not have an associated gesture. Two solutions: -, a button in the dialog allows him to activate the script. - he must use another dialog to activate it. I think that apart from a technical problem, the choice between the two solutions is obvious.Even if you are replying to my message https://github.com/nvaccess/nvda/issues/17209#issuecomment-2376655053, you seem to totally ignore the virtual interface proposal.
Have you tested Command Helper add-on? If yes, why have you preferred the dialog solution in your add-on? You have surely had good reasons, but it would be interesting to make them all explicit here so that they are shared with everyone.
Re the choice between input gesture dialog and another dialog, again explicit is better than implicit. So what seem obvious to me is that if we go forward with a dialog solution, your solution is the best one, i.e. add one more button in the input gesture dialog rather than creating a brand new dialog.
— Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/17209#issuecomment-2377371489, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZLFFAQWDEONBKFUPAJA2LZYQV2TAVCNFSM6AAAAABOXNJE4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZXGM3TCNBYHE. You are receiving this because you were mentioned.Message ID: @.***>
Hello. If nobody creates a PR for this, I may try to do it. I haven't worked on this in an add-on, so I'm not sure if I would be able to do it successfully. I'm offering help with this since for now nobody has created a PR, and I'd like that new users and people with dexterity or cognitive issues, and all, have this included in NVDA.
There are two design proposals for this feature:
- a visible dialog with a specific button to activate the command, as done in NVDA global commands extension (Paul's add-on) or Virtual Palette (Blindpandas' add-on)
- a virtual interface with nothing displayed on the screen, as done in Command Helper (Javi's add-on) or Voice Over
The pros and cons of these two options have been discussed here. It would be nice that NV Access make the final design choice. @SaschaCowley, @seanbudd what would be NV Access choice between these 2 designs? Note that both could also be accepted so that the commands are more easily discoverable and reachable (dialog option), but the commands can be executed also without moving the focus/navObject (virtual interface option).
Also, multiple-presses should be supported IMO.
Imo, it would be desirable that commands are vissible, for people with low vission and to facilitate learning them vissually too. I would prefer to work enhancing the keyboard gestures dialog, leveraging the existing dialog. A context menu may be created for repeated gestures, i.e., when scriptCount is used.
For information, my extension adds, in the "Input gestures" dialog, an "activate script" button. To execute a script, the “executeScript” method of the “scriptHandler” module with a repeat counter is used. When the cursor is on the button:
- pressing the "enter" key closes the dialog and starts execution of the script with the counter at 0.
- pressing the "space" key starts the timeout for multiple presses. During this delay, pressing the “space” key again increments the repetition counter and restarts the delay. At the end of this period, the dialog is closed and execution of the script is launched with the current value of the repetition counter as a parameter.
Another modification made in this dialog is to automatically select, if possible, the application scripts category under the cursor and expand it.
Le 07/10/2024 11:51, Noelia Ruiz Martínez a écrit :
Imo, it would be desirable that commands are vissible, for people with low vission and to facilitate learning them vissually too. I would prefer to work enhancing the keyboard gestures dialog, leveraging the existing dialog. A context menu may be created for repeated gestures, i.e., when scriptCount is used.
— Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/17209#issuecomment-2396449473, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZLFFDUTRTYNSY4ALVNMILZ2JKRLAVCNFSM6AAAAABOXNJE4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJWGQ2DSNBXGM. You are receiving this because you were mentioned.Message ID: @.***>
@CyrilleB79
- a virtual interface with nothing displayed on the screen, as done in Command Helper (Javi's add-on) or Voice Over
Just FYI, VO's menus are visible on screen.
I think this is likely to work most reliably if we don't change focus. And it will have the most utility if it is visible on screen (and ideally supports mouse input). One possibility I have thought about is creating an always on top but non-focused window which presents the GUI, and sending the appropriate wx events to it as the user navigates through the virtual command palette layer in NVDA. That way we leave the majority of the GUI to wx, while not modifying NVDA's focus. I am reasonably sure this would be possible, it may not be practical though. And I don't think it would support mouse input, as clicking the controls would move NVDA's focus.
I think that we can use the alternative suggested by @SaschaCowley :
Add an "Activate" button to the input gestures dialog, which performs the selected script in the context from which the user opened the input gestures dialog.
Perhaps we can use the scriptInfo importing dynamically the corresponding module with importlib.import_module, using gui.mainFrame.prevFocus.setFocus() once the activate command button has been pressed and the dialog has been destroyed, and then we may use scriptHandler.executeScript(importedModule.className.script, gesture). This is theory, unless someone has tried to use this in an add-on or something. I'll try to test this locally in the next days, unless someone has any concerns or suggestions.
@nvdaes, as written before, the activate button solution is implemented in Paul's add-on; you may want to have a look.
Though, even if it applies the command to previous focus, this solution moves the focus. As explained by @SaschaCowley, it may not be desirable in certain cases. For example, if you have a command that you want to execute while a context menu is open, with the solution in the input gesture dialog, the dialog will pop up, closing the menu; then, pressing "Activate" button, the dialog will be closed and the focus will be restored, but the context menu will not be reopened. That's why I am supporting the "virtual interface" solution. Or better, the non focusable interface as described by @SaschaCowley.
@SaschaCowley proposal about a non focusable interface, according by her, has also drawbacks. And I see the disadvantage using the keyboard gestures dialog, for example if a menu is opened. For now I won't take this since I don't know how to do it in a satisfactgory way. Hope this can be done.
@nvdaes could you be more specific with the drawbacks that you have in mind regarding the non focusable interface (@SaschaCowley's proposal)?
The only one that I have in mind is that mouse input may not be supported, or at least not easily implementable.
could you be more specific with the drawbacks that you have in mind regarding the non focusable interface
In addition to the mouse drawback, I don't find comfortable to use, for example, the emoji panel to search emojis with suggestions. Also, to read the active non focusable element maybe needed to use commands to report the current object or similar, and the report line command may read the focused element. This is not comfortable for me, and when pressing Escape some kind of notification may be needed to be sure that the non focusable gui is closed. For me is much more comfortable a conventional dialog like the input gestures, but I understand that it wouldn't be appropriate to be used for example in menus.
There may be another alternative solution: Start the execution of a script without a input gesture using another script with a dedicated input gesture. In the "Input gestures" dialog, a button would allow the user to save the selected script for later execution. Thus the user would have the possibility, without having to assign a input gesture, to launch any script using the dedicated input gesture, one or more times without needing to return to the "Input gestures" dialog or any virtual dialogue.
Le 08/10/2024 23:28, Cyrille Bougot a écrit :
@nvdaes https://github.com/nvdaes, as written before https://github.com/nvaccess/nvda/issues/17209#issuecomment-2370329873, the activate button solution is implemented in Paul's add-on; you may want to have a look.
Though, even if it applies the command to previous focus, this solution moves the focus. As explained https://github.com/nvaccess/nvda/issues/17209#issuecomment-2398031553 by @SaschaCowley https://github.com/SaschaCowley, it may not be desirable in certain cases. For example, if you have a command that you want to execute while a context menu is open, with the solution in the input gesture dialog, the dialog will pop up, closing the menu; then, pressing "Activate" button, the dialog will be closed and the focus will be restored, but the context menu will not be reopened. That's why I am supporting the "virtual interface" solution. Or better, the non focusable interface as described https://github.com/nvaccess/nvda/issues/17209#issuecomment-2398031553 by @SaschaCowley https://github.com/SaschaCowley.
— Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/17209#issuecomment-2400853677, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZLFFBUM6SI4EBND7QV5PDZ2RE7VAVCNFSM6AAAAABOXNJE4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBQHA2TGNRXG4. You are receiving this because you were mentioned.Message ID: @.***>
Í like a lot the alternative of saving commands to be run later with a dedicated gesture, combining it with the abilite to run them directly, giving both alternatives. If someone with more experience can create a PR, it would be great. Otherwise I may do it.
I'm working on a local branch for this feature. I've consulted commandHelper by Javi, available on the store, and GlobalExtensions, cloned from
https://github.com/paulber19/NVDAExtensionGlobalPlugin
My approach for now is different respect both add-ons, though the source code of both add-ons is interesting and useful. For now I have a Run button which executes scripts after closing the Input Gestures dialog. Sincerely I would prefer to create commands to avoid using repeated gestures from the palette. I think that this is not intuitive and it's intended to be used by pressing keystrokes, not from a dialog. Also, for now I'm not using cancelSpeech since imo there is no reason to suppress the feedback when the dialog is closed. Anyway I'm thinking about using speech cancellation to report the focused object with a short message before executing the script. Also, I'm thinking about character navigation in browse mode. I'm not sure if this should be included in the palette, since using arrows maybe more practical if the purpose is to navigate elements in a quick way. But I'll see all this and I'll consider feedback.
I've linked an initial draft PR. I'll improve its description, for example mentioning existing add-ons and suggestions. This is in a very initial stage.
Hello, I'll mark my PR as ready to review, to know if the general design will be accepted or not. I've mentioned commandHelper and globalExtension add-ons. Please, let me know if more detailed info about these add-ons should be added in the pull request.
@seanbudd , @SaschaCowley , since PR #17290 is not considered appropriate, and it's preferred a virtual window, I'll wait until this is implemented by other person and then I'll try to review and test the created PR. The reason is that NVDA doesn't have virtual windows and I don't know how this should be created, and if a different dependency like QT or any other should be required.