kolibri icon indicating copy to clipboard operation
kolibri copied to clipboard
verified publisher icon learningequality

Crowdin API v1 to v2 & Message extraction fixes

Open nucleogenesis opened this issue 3 years ago • 1 comments

Summary

Updates the Crowdin script significantly to:

  • Implement the v2 API for required commands (Upload sources, Download translations, Upload/Download Glossaries, Pre-translate, Build Translations)
  • Make it easy to add features as needed (looking at you screenshots)
  • Adds some helpful commands to fetch information from the API as needed (list project, glossaries, files for branch)

Fixes bugs in message extraction:

  • Aliased imports failing in studio
  • Messages using a backticks but without any interpolation were being skipped because thats a Literal node and not a TemplateLiteral node...

Does not port over from old script:

  • Upload translations. @radinamatic noted this was only really used for hacky testy purposes. Adding it if/when needed should be okay
  • Stats. No comparable API endpoint in v2.

API Module Approach

We use the Crowdin API Python Client for API requests.

I created a module lib.i18n.crowdin which includes 4 packages api, cli, config, utils to encapsulate some bits of the logic.

lib.i18n.crowdin.config

This file declares a dict configuration which is used to generate the commands accepted by argsparse.

Each configuration item provides:

  • func: A function (one of the ones defined in lib.i18n.crowdin.api)
  • required_args: A list of snake_case arguments (those configured in lib.i18n.crowdin.cli)
  • help: A message explaining the command.

I believe that this is flexible enough for what we need. The required_args can take a string OR an array of strings. When it finds an array, it only requires that one of those be provided.

This also provides a text fixture of a validly shaped and mocked configuration dict.

lib.i18n.crowdin.cli

Defines argsparse and provides some locally significant helper functions around handling of CLI input. Parses the configuration, calls the right function given the commands, creates a consistent set of options to be passed along to API functions.

lib.i18n.crowdin.api

Where the API calls are made and where the functions imported by the config.

lib.i18n.crowdin.utils

Mostly old utils here still useful, added a few more for handling all potential "where to find the files" logic depending on if I'm working only with a path, a file listing Python modules or user-passed list of Python modules.

References

Fixes #9678

Reviewer guidance

@rtibbles let's find a time to walk through together - feel free to invite a +1 :)

nucleogenesis avatar Sep 15 '22 22:09 nucleogenesis

Build Artifacts

Asset type Download link
PEX file kolibri-0.16.0.dev0_git.20221011213040.pex
Unsigned Windows installer kolibri-0.16.0.dev0+git.20221011213040-unsigned.exe
Debian Package kolibri_0.16.0.dev0+git.20221011213040-0ubuntu1_all.deb
Mac Installer (DMG) kolibri-0.16.0.dev0+git.20221011213040-0.3.0.dmg
Source Tarball kolibri-0.16.0.dev0+git.20221011213040.tar.gz
WHL file kolibri-0.16.0.dev0+git.20221011213040-py2.py3-none-any.whl

github-actions[bot] avatar Sep 15 '22 22:09 github-actions[bot]

@rtibbles successfully have tested this on Kolibri as well as Studio should be ready for final review

nucleogenesis avatar Sep 30 '22 18:09 nucleogenesis

While routinely testing the build assets for this one I noticed that both exercises and html resources cannot be viewed. Just flagging it.

HTML5:

https://user-images.githubusercontent.com/79847249/215529161-a0e88738-b221-416a-beea-cf0c6192450f.mp4

EXERCISES:

https://user-images.githubusercontent.com/79847249/215529190-28ecdf59-db72-4b3a-a6df-2c72ab2f395c.mp4

pcenov avatar Jan 30 '23 16:01 pcenov

Testing the latest built DEB asset from the develop branch on Ubuntu, I can confirm both of the issues, so they are not caused by this PR. @pcenov could you please open a separate issue for these? Thank you!

2023-01-30_22-24-04

One interesting thing: HTML resources do seem to fail to open, but after some time passes, completion is achieved, and learner can see the modal.

radinamatic avatar Jan 30 '23 21:01 radinamatic

maybe we should schedule a session w/ @radinamatic to walk through

That would be most appreciated 🙏🏽

radinamatic avatar Jan 30 '23 22:01 radinamatic

~@nucleogenesis~ rtibbles is on it I bumped into Error: Invalid value for "BRANCH": Branch is not valid error. I've put apiv2 branch name just to test and delete later. Branch names should be able to contain both letters and numbers, so far they did, at least, correct?

$ make i18n-upload branch=apiv2
yarn run makemessages
yarn run v1.17.3
$ kolibri-tools i18n-extract-messages --pluginFile ./build_tools/build_plugins.txt
Gathering relevant modules from ['kolibri.core', 'kolibri.plugins.*']
Writing webpack_json output to /tmp/202312-10701-y0z9eh.282y.json
INFO: Writing webpack_json output to /tmp/202312-10701-y0z9eh.282y.json
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/core/**/*.@(vue|js)
Kolibri: Processing  227  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/core/**/*.@(vue|js)
Kolibri: Processing  227  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/user_profile/**/*.@(vue|js)
Kolibri: Processing  39  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/user_profile/**/*.@(vue|js)
Kolibri: Processing  39  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/media_player/**/*.@(vue|js)
Kolibri: Processing  32  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/facility/**/*.@(vue|js)
Kolibri: Processing  71  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/facility/**/*.@(vue|js)
Kolibri: Processing  71  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/epub_viewer/**/*.@(vue|js)
Kolibri: Processing  32  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/user_auth/**/*.@(vue|js)
Kolibri: Processing  30  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/user_auth/**/*.@(vue|js)
Kolibri: Processing  30  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/coach/**/*.@(vue|js)
Kolibri: Processing  243  files...
Kolibri: Skipping /home/didivm/LE/kolibri/kolibri/plugins/coach/assets/src/views/common/LearnerExerciseReport.vue because $trs property was given an empty object
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/coach/**/*.@(vue|js)
Kolibri: Processing  243  files...
Kolibri: Skipping /home/didivm/LE/kolibri/kolibri/plugins/coach/assets/src/views/common/LearnerExerciseReport.vue because $trs property was given an empty object
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/setup_wizard/**/*.@(vue|js)
Kolibri: Processing  51  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/html5_viewer/**/*.@(vue|js)
Kolibri: Processing  4  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/slideshow_viewer/**/*.@(vue|js)
Kolibri: Processing  4  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/demo_server/**/*.@(vue|js)
Kolibri: Processing  3  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/perseus_viewer/**/*.@(vue|js)
Kolibri: Processing  39  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/learn/**/*.@(vue|js)
Kolibri: Processing  170  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/learn/**/*.@(vue|js)
Kolibri: Processing  170  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/pdf_viewer/**/*.@(vue|js)
Kolibri: Processing  14  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/device/**/*.@(vue|js)
Kolibri: Processing  123  files...
Kolibri: Getting files from glob: /home/didivm/LE/kolibri/kolibri/plugins/device/**/*.@(vue|js)
Kolibri: Processing  123  files...
Kolibri: Removing existing messages files from /home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES
Kolibri: Successfully cleared path for CSVs by removing: /home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.core.default_frontend-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.core.frontend_head_assets-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.coach.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.coach.side_nav-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.demo_server.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.device.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.device.side_nav-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.epub_viewer.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.facility.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.facility.side_nav-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.html5_viewer.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.learn.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.learn.side_nav-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.media_player.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.pdf_viewer.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.perseus_viewer.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.setup_wizard.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.slideshow_viewer.main-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.user_auth.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.user_auth.login_side_nav-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.user_profile.app-messages.csv
/home/didivm/LE/kolibri/kolibri/locale/en/LC_MESSAGES/kolibri.plugins.user_profile.user_profile_side_nav-messages.csv
Kolibri: Successfully extracted 3574 messages!
Kolibri: Messages successfully written to CSV files.
Done in 37.72s.
cd kolibri && python -m kolibri manage makemessages -- -l en --ignore 'node_modules/*' --ignore 'kolibri/dist/*'
INFO     2023-02-02 11:25:47,404 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2023-02-02 11:25:47,405 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO     2023-02-02 11:25:47,405 Option RUN_MODE in section [Deployment] being overridden by environment variable KOLIBRI_RUN_MODE
INFO     2023-02-02 11:25:47,734 Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     2023-02-02 11:25:47,998 Invoking command makemessages -l en --ignore node_modules/* --ignore kolibri/dist/*
processing locale en
python packages/kolibri-tools/lib/i18n/crowdin.py upload-sources apiv2
INFO: Project 'Kolibri - Learning Platform' found on Crowdin with ID '201933'
Branch apiv2 was not found. Would you like to create that branch? [y/N]: y
Usage: crowdin.py upload-sources [OPTIONS] BRANCH

Error: Invalid value for "BRANCH": Branch is not valid
Makefile:231: recipe for target 'i18n-upload' failed
make: *** [i18n-upload] Error 2

radinamatic avatar Feb 02 '23 19:02 radinamatic

It seems I had 'yes' meaning 'no' and 'no' meaning 'yes' - which is liable to confuse anyone!

rtibbles avatar Feb 02 '23 19:02 rtibbles