keymanapp
epic: new gesture engine 🐵
Implements #5029.
Originally from #6842:
By developing an isolated module for gesture support, we'll be able to demo and test gesture behavior outside of KeymanWeb. One of my design goals with this is actually to enable gesture unit tests by recording tracked input sequence data. This may be achieved by keeping the core logic headless and replicating recorded input sequences in a headless environment.
This branch should not be merged into master until the new module is sufficiently ready and has been integrated into KeymanWeb properly. Until then, it should live as a feature branch.
The following breakdown will be updated as development on the feature continues.
Phase 1: the base GestureRecognizer
This section will be devoted to establishing the interface for the basic infrastructure needed to properly support gestures given the constraints needed by KeymanWeb and ensuring that future work on this module will be testable independently of Web.
Goals:
- a host page that can replicate the needs of a touch-platform OSK, including screen edge limitations, etc.
- Such a page will be quite useful for demoing gesture behavior without the need to worry about how it will affect Web.
- Either the same page or an extended page that allows recording of inputs received, for use in unit-test development (much like
web/tools/recorder's role for KeymanWeb's engine tests) - The ability to support tracking of multiple touchpoints.
PRs for this phase:
- #6842
- For future reference: through commit https://github.com/keymanapp/keyman/pull/6842/commits/d5222dfaadf94989b5c230a863cf81ffbd73a1bd.
- It was the original 'feature branch' before we set up this one as a holding ground instead.
- #6843
- #6844
- #6878
- Hooray for something interactive!
- Has an initial set of user tests.
- #6914
- #6952
- Adds an input-sequence recorder page useful for unit-test preparation
- #6970
- Actual unit tests!
Phase 2: the GestureSegmenter
This section will be devoted to analyzing raw input-event data and mapping those time-based input sequences to gesture "segments" - small-scale gesture components that will serve as "building blocks" for complex gesture types, their differentiation, and state modeling.
- #7058
- #7142
- #7157
-
Possibly one more (for 'integration'-style, input-event through to segmentation tests)
- There's a strong chance I'll skip this aspect for now unless requested otherwise.
- They're definitely possible to write, but this feature's implementation is running slower than originally projected.
Phase 3: the GestureSynthesizer
Once the "building block" logic is sound and stable, we may then use those components to construct the complex gestures needed by KeymanWeb's OSK and the internal state machine(s) needed for differentiation, detection, recognition, cancellation, and resolution.
- #7440
- Contains a unit test with extremely early prototyping for FSM-like processing of the segmentation layer's async outputs.
- TBD.
Phase 4: Keyman Engine for Web integration
Once the module is ready to go, we need to integrate it with our Web engine's OSK.
Related Issues
- [ ] https://github.com/keymanapp/keyman/issues/7067
@keymanapp-test-bot skip
User Test Results
Test specification and instructions
- ✅ TEST_REGRESSION_WEB (PASSED): I ran the full regression test suite for the web (except SUITE_TOUCH_GESTURES section) with the given "KeymanWeb Test Home" test page and here is my observation: 1. Tested in all Desktop Platforms (Windows_Firefox, Chrome, MacOS_Safari, Chrome and Firefox, Linux_Firefox) and it seems to be working as expected. No critical issues were found during my testing. 2. Additionally, I conducted tests on touch platforms such as Android, iPhone, and iPad using the specified browsers and configurations, and they appear to be functioning properly.
- 🟥 TEST_REGRESSION_ANDROID_KBD (FAILED): Retested with the attached PR build (Keyman 17.0.227-alpha-test-7324 in the Android Mobile device (ver 13) and here is my observation: 1. While testing TEST_INAPP_LATIN_KEYBOARD_LANDSCAPE test in the lanscape mode the OSK shrinks it width after using the globe key. Here, What I did is : (notes)
- 🟥 TEST_REGRESSION_IOS_PREDTEXT (FAILED): Run the regression test for iOS (iPhone 13 Mobile) (SUITE_PREDICTIVE_TEXT_AND_AUTO_CORRECTION) and here is my observation: 1. Installed the attached PR build Keyman 17.0.221-alpha-test-7324 using TestFlight. 2. It seems that TEST_BROKEN_SUGGESTION, TEST_OP, TEST_NEW_Q, TEST_E_P and TEST_GUILLEMET are passed. 2. However, TEST_QUOTE_O and TEST_QUOTE_OP are failed. I have attached the Screenshots for reference. (notes)
Retesting Template
@keymanapp-test-bot retest TEST_REGRESSION_ANDROID_KBD TEST_REGRESSION_IOS_PREDTEXT
Test Artifacts
- Android
- Developer
- iOS
- Keyboards
- Web
- Windows
![keymanapp-test-bot[bot] avatar](https://avatars.githubusercontent.com/in/133554?v=4 "Published by a pub.dev verified publisher"){kind=small}
Note https://github.com/keymanapp/keyman/pull/7741#issuecomment-1322875467 when it comes time to merge; I originally did some test index reorganization, but there were a few hrefs that needed adjustment.
Related: https://github.com/keymanapp/keyman/issues/5511
It'll be lower-priority to implement than the main gesture types we've already added in Developer (multi-tap, output-key flick), but we can probably implement it in the same release cycle.
if the mutlitap feature is available in 17.0.36 (0.7324.7695),
If I can get the keyboard format. I can test also.
if the mutlitap feature is available in 17.0.36 (0.7324.7695),
If I can get the keyboard format. I can test also.
- The gesture work in the apps is still in process, not yet ready to test.
- The gesture design tools are available in Keyman Developer 16.0.138.
No pressure, Just curious if the gesture engine ready for testing
Not yet.
Test Results
- TEST_REGRESSION_WEB (PASSED): I ran the full regression test suite for the web (except SUITE_TOUCH_GESTURES section) with the given "KeymanWeb Test Home" test page and here is my observation: 1. Tested in all Desktop Platforms (Windows_Firefox, Chrome, MacOS_Safari, Chrome and Firefox, Linux_Firefox) and it seems to be working as expected. No critical issues were found during my testing. 2. Additionally, I conducted tests on touch platforms such as Android, iPhone, and iPad using the specified browsers and configurations, and they appear to be functioning properly.
Test Results
- TEST_REGRESSION_ANDROID_KBD (FAILED): Run the regression test for Android (Suite_Keyboard_Functionality) in the Android Mobile (ver 13)and here is my observation: 1. Tested (TEST_INAPP_LATIN_KEYBOARD_PORTRAIT) and I am getting keyboard error message while using backspace, enter and space keys during the testing. Seems to be an issue.
..Error in keyboard
..video file
https://github.com/keymanapp/keyman/assets/19683143/c820bf67-14cc-4294-8c0a-27af65f602e0
Test Results
- TEST_REGRESSION_IOS_PREDTEXT (FAILED): Run the regression test for iOS (iPhone 13 Mobile) (SUITE_PREDICTIVE_TEXT_AND_AUTO_CORRECTION) and here is my observation: 1. Installed the attached PR build Keyman 17.0.221-alpha-test-7324 using TestFlight. 2. It seems that TEST_BROKEN_SUGGESTION, TEST_OP, TEST_NEW_Q, TEST_E_P and TEST_GUILLEMET are passed. 2. However, TEST_QUOTE_O and TEST_QUOTE_OP are failed. I have attached the Screenshots for reference.
..Test_quote_O
..Test_quote_OP
Test Results
* **TEST_REGRESSION_ANDROID_KBD (FAILED):** Run the regression test for Android (Suite_Keyboard_Functionality) in the Android Mobile (ver 13)and here is my observation: 1. Tested (TEST_INAPP_LATIN_KEYBOARD_PORTRAIT) and I am getting keyboard error message while using backspace, enter and space keys during the testing. Seems to be an issue.[omitted]
..video file ErrorinKeyboard.mp4
Timestamp in video is ~1:36, after coming out of the language-picker. Just noting, since the video is not isolated to the error itself.
The error was logged to Sentry and matches #10136's events. Therefore, it's at least a known issue, though an exact repro isn't known quite yet.
Test Results
- TEST_REGRESSION_IOS_PREDTEXT (FAILED): Run the regression test for iOS (iPhone 13 Mobile) (SUITE_PREDICTIVE_TEXT_AND_AUTO_CORRECTION) and here is my observation: 1. Installed the attached PR build Keyman 17.0.221-alpha-test-7324 using TestFlight. 2. It seems that TEST_BROKEN_SUGGESTION, TEST_OP, TEST_NEW_Q, TEST_E_P and TEST_GUILLEMET are passed. 2. However, TEST_QUOTE_O and TEST_QUOTE_OP are failed. I have attached the Screenshots for reference.
..Test_quote_O
..Test_quote_OP
Huh. I can't repro those screenshots... but even then, the suggestions are, at least, pretty related. For the second one, in particular, all the suggestions do include op within them, even if not word-initial. I'm not too worried about this one, though I am curious how those specific suggestions ended up winning out.
I do wonder if this is related to the known issue #10040; perhaps a state like that would affect things. But... the easiest known way to cause that state isn't viable... to my knowledge... with sil_euro_latin. 🤷
I just happened to catch two new errors worth addressing:
- #10170
- Probably pretty "minor", all things considered
- #10171
- This one, however, is not minor, as it can more easily trigger noticeable errors - and Sentry error events.
In association with the previously-noted #10136, that makes three total known gesture issues that currently remain.
- #10136
TODO:
- KMW's
build.shhas a "temporary" tie-in component for the gesture engine. We should be able to drop that in favor of plain build dependencies now.- That said, the test-page copy stuff should be migrated to the test-resource copying shell-script.
@keymanapp-test-bot retest TEST_REGRESSION_ANDROID_KBD
#10136 should be fixed by #10185 now, and I believe that was the reason for the previously-failing user test.
iOS build failure from the last commit:
07:51:22
/Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/FirstVoices.xcodeproj: error: No signing certificate "iOS Distribution" found: No "iOS Distribution" signing certificate matching team ID "D7TR486TEH" with a private key was found. (in target 'SWKeyboard' from project 'FirstVoices')
07:51:22
/Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/FirstVoices.xcodeproj: error: Provisioning profile "FirstVoices_Keyboards_SWKeyboard" expired on 10 Dec 2023. (in target 'SWKeyboard' from project 'FirstVoices')
07:51:22
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'SWKeyboard' from project 'FirstVoices')
07:51:22
/Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/FirstVoices.xcodeproj: error: No signing certificate "iOS Distribution" found: No "iOS Distribution" signing certificate matching team ID "D7TR486TEH" with a private key was found. (in target 'FirstVoices' from project 'FirstVoices')
07:51:22
/Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/FirstVoices.xcodeproj: error: Provisioning profile "FirstVoices_Keyboards" expired on 10 Dec 2023. (in target 'FirstVoices' from project 'FirstVoices')
07:51:22
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'FirstVoices' from project 'FirstVoices')
07:51:22
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'FirstVoices' from project 'FirstVoices')
07:51:22
** ARCHIVE FAILED **
07:51:22
07:51:22
07:51:22
[oem/firstvoices/ios] Build failed! Error: [65] when executing command: 'xcodebuild -quiet -configuration Release -derivedDataPath "/Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/build" -scheme FirstVoices -archivePath /Users/mcdurdin/buildAgent/work/99b311828f4ee7c/keyman/oem/firstvoices/ios/build/Build/Products/Release-iphoneos/FirstVoices.xcarchive archive -allowProvisioningUpdates VERSION=17.0.227 VERSION_WITH_TAG=17.0.227-alpha-test-7324'
07:51:22
07:51:22
[oem/firstvoices/ios] ## build failed
Provisioning profile "FirstVoices_Keyboards_SWKeyboard" expired on 10 Dec 2023.
For the FirstVoices subproject, as built by ba-macos-ext-hba-03.
For the FirstVoices subproject, as built by
ba-macos-ext-hba-03.
merge master into your branch
merge master into your branch
and I see now that this is a separate failure ... opening as a new issue
Test Results
- TEST_REGRESSION_ANDROID_KBD (FAILED): Retested with the attached PR build (Keyman 17.0.227-alpha-test-7324 in the Android Mobile device (ver 13) and here is my observation: 1. While testing TEST_INAPP_LATIN_KEYBOARD_LANDSCAPE test in the lanscape mode the OSK shrinks it width after using the globe key. Here, What I did is :
- In the Landscape mode, typed some letter using the OSK.
- Used Backspace and Enter button after typing each words.
- Clicked the globe key.
- Selected the same keyboard from the Picker menu.
Here, I noticed that the OSK shrinks its width on the Keyman home page. Seems to be an issue.
Test Results
* **TEST_REGRESSION_ANDROID_KBD (FAILED):** Retested with the attached PR build (Keyman 17.0.227-alpha-test-7324 in the Android Mobile device (ver 13) and here is my observation: 1. While testing TEST_INAPP_LATIN_KEYBOARD_LANDSCAPE test in the lanscape mode the OSK shrinks it width after using the globe key. Here, What I did is : 1. In the Landscape mode, typed some letter using the OSK. 2. Used Backspace and Enter button after typing each words. 3. Clicked the globe key. 4. Selected the same keyboard from the Picker menu.Here, I noticed that the OSK shrinks its width on the Keyman home page. Seems to be an issue.
Documented as #10241. I have reproduced this on master - thus it is not a feature-gestures issue. There's a chance this is related to recent ✨ Android optimizations.
At any rate... it's not something to block upon.
