openedx
docs: ADR introducing mobile offline content support
Description
This is draft ADR that describes offline content generation to make it available on mobile.
Supporting information
FC-0047 (link TBD)
Deadline
06/28/2024
Thanks for the pull request, @GlugovGrGlib!
This repository is currently maintained by @openedx/wg-maintenance-edx-platform.
Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.
:radio_button: Get product approval
If you haven't already, check this list to see if your contribution needs to go through the product review process.
- If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
- If it doesn't, simply proceed with the next step.
:radio_button: Provide context
To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
- Dependencies
This PR must be merged before / after / at the same time as ...
- Blockers
This PR is waiting for OEP-1234 to be accepted.
- Timeline information
This PR must be merged by XX date because ...
- Partner information
This is for a course on edx.org.
- Supporting documentation
- Relevant Open edX discussion forum threads
:radio_button: Get a green build
If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.
Where can I find more information?
If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:
When can I expect my changes to be merged?
Our goal is to get community contributions seen and reviewed as efficiently as possible.
However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
- The size and impact of the changes that it introduces
- The need for product review
- Maintenance status of the parent repository
:bulb: As a result it may take up to several weeks or months to complete a review and merge your PR.
I suggest adding a comprehensive list of content types that will support offline viewing (maybe not all will be implemented at a time). I think that video content deserves special attention. Many courses rely on video content. Open edX usually relies on external streaming services for video (YouTube, Vimeo, etc.) Video files can be huge. How will it handle offload video? Maybe uploading a downloadable version of the video in a separate storage? Another thing to consider is milestones. If grading cannot be done offline, then probably unlocking content based on subsection grade will not be possible.
I suggest adding a comprehensive list of content types that will support offline viewing (maybe not all will be implemented at a time). I think that video content deserves special attention. Many courses rely on video content. Open edX usually relies on external streaming services for video (YouTube, Vimeo, etc.) Video files can be huge. How will it handle offload video? Maybe uploading a downloadable version of the video in a separate storage? Another thing to consider is milestones. If grading cannot be done offline, then probably unlocking content based on subsection grade will not be possible.
Hello @angonz! I have updated the ADR and now there is a list of blocks that are planned to be supported. I also added a diagram to make it clearer how the content generation process will work. Please check it out.
As for videos, video uploading is already implemented in mobile applications. As for unlocking content based on subsection grade, yes, such functionality is not planned within this work scope and, accordingly, it is not reflected in the ADR.
@GlugovGrGlib Just for curiosity how this ADR is different from this PR?
Hi @salman2013, the functionality added in the PR you mentioned corresponds to what is described in this ADR. This ADR describes the overall vision, but focuses on how it is planned to be done on the backend.
@kdmccormick and I had a chance to talk through some of the details in his review. I agree that adding a view to XBlocks to support this new case is a superior approach to introducing a new facility to render XBlocks that is outside of the design we currently have.
I'm not as convinced on the duplicate asset question as I think that the transfer and storage cost of problem blocks is probably a rounding error compared to video. We should do some analysis of a highly complex course to see what the impact would be. We are trading off exported blocks being self-sufficient against optimizing transfer and storage. All to say, I think more discovery is required here.
We also discussed that even with XBlocks being able to advertise whether they support offline mode that we probably need a site-wide configuration option that allows operators to allow/deny list which blocks they want available on mobile. This would be new scope here.
It's also worth noting that because the problem block wraps different things, some of which will be offline ready and some of which will not, that there will be some messiness in the internals of the block. This would also complicate implementing the site-wide config because an operator might want to allow some types of problem renderings and not others.
As finalizing this is blocking closing our a project, I recommend that we close the project, leave this ADR open, and come back to it when offline mode is back on the frontlog.
@kdmccormick Thank you for your detailed response and patience. I have added part of this description to ADR 1 and put the rest of the future improvements in a separate ADR 2.
Hello @kdmccormick! Here are answers on questions:
Please clarify that this offline content generation happens in the CMS process, not LMS.
Currently, offline content generation is handled inside the LMS process. The CMS sends a publish signal (course_cache_updated), which triggers a Celery task that lives in LMS and renders content using the LMS-based XBlockRenderer. Since block content in CMS is currently rendered using legacy backbone JS, a huge number of JS files would need to be included in the block archive, and mobile devices would then have to work with them. In the future, we may consider shifting the full generation pipeline to CMS to reduce cross-system complexity.
I'm glad that de-duping of assets is now in the plan. But, I'm skeptical that we should be generating a zip archive for the entire course. This would cause the regeneration of the entire zip each time any block in the course is edited. Could you explain the decision to zip the entire course rather than zip at a lower level, like the Unit?
We do not make one big ZIP for the whole course. Instead, we build and store a ZIP file for each block (e.g. each problem or text unit). When you update a block, only that block’s ZIP is re‑created, not the entire course.
The view methods of the structural blocks (course, section/chapter, and subsection/sequential) are not used any more since legacy courseware and legacy Studio have been deprecated. The medium-term direction is actually to stop treating those as a XBlocks. So, I do not think we should add offline_view to them. Will you still need to pre-render those structures, or would it work to only pre-render components (problems, text, etc.), and assume that the app has already downloaded the structure of the course?
We don’t plan to add offline_view to structural blocks like course, chapter, or sequential. These blocks don’t contain content themselves and are used only to organize other blocks. The course structure can be provided separately, so we only pre-render content blocks like problems, HTML, and videos.
The text of the ADR does a good job explaining what offline_view is and how it fits into the system. Could you edit the diagram to also include of
offline_view?
Currently I'm uploading updated diagram. I also extended CMS part a bit with course cache updates process.
@kyrylo-kh
We do not make one big ZIP for the whole course. Instead, we build and store a ZIP file for each block (e.g. each problem or text unit). When you update a block, only that block’s ZIP is re‑created, not the entire course.
Sounds good.
We don’t plan to add offline_view to structural blocks like course, chapter, or sequential. These blocks don’t contain content themselves and are used only to organize other blocks. The course structure can be provided separately, so we only pre-render content blocks like problems, HTML, and videos.
Perfect!
Currently, offline content generation is handled inside the LMS process. The CMS sends a publish signal (course_cache_updated), which triggers a Celery task that lives in LMS and renders content using the LMS-based XBlockRenderer.
I still feel strongly that CMS should be pre-rendering these ZIP archives rather than the LMS. The pre-rendering is part of the "publish" stage of the content lifecycle and does not involve student state, so it conceptually belongs with the CMS. From a practical standpoint, handling CMS events in the LMS is not something we have done before... doing this would introduce new failure modes and create unnecessary coupling between CMS and LMS.
Since block content in CMS is currently rendered using legacy backbone JS, a huge number of JS files would need to be included in the block archive, and mobile devices would then have to work with them. In the future, we may consider shifting the full generation pipeline to CMS to reduce cross-system complexity.
Are you referencing the studio_view (the view that renders the editors)? We would not be pre-rending the studio_view. The offline_view would be more similar to the author_view, which is very similar to the student_view that the LMS displays. I do not know of any additional legacy backbone JS dependencies necessary to render the author_views of problem, html, or video blocks. Could you point me to some code that shows how CMS would need these additional dependencies in order to pre-render blocks?
Hey @kyrylo-kh , can you give me a rough estimate on when you'll be working on this again, just so I can make sure I can carve out some time to give a prompt review and help get it merged?
Hello @kdmccormick We’ve scheduled the investigation to reproduce the offline content rendering issue in CMS and will follow up with an estimate and proposed next steps in approximately one week.
@kdmccormick
At the moment the ZIP export uses the LMS courseware-chromeless.html LMS template. This template already includes the right LMS JS/CSS and renders correctly on its own. That is why the ZIP works fine when generated from the LMS.
I tried moving the rendering into the CMS, but the closest template there is container_chromeless.html. This one is very different: it bootstraps Studio editing UI and loads container.js, which pulls in Backbone/Underscore views and many Studio-only dependencies. Example:
<script src="assets/bundles/js/factories/container.js" type="text/javascript"></script>
<script type="text/javascript">
ContainerFactory(
[{"type": "html", ...}],
"view",
{
isUnitPage: false,
canEdit: true,
outlineURL: "/course/course\u002Dv1:offline+1+1?format\u003Dconcise",
clipboardData: {"content": null, "source_usage_key": "", "source_context_title": "", "source_edit_url": ""},
isIframeEmbed: true,
}
);
</script>
That code is meant for authoring, not for student/offline use. It brings a whole chain of legacy modules that are hard to run in a simple offline archive.
To realistically support this in the CMS we would need to:
- write and maintain a new template specifically for offline rendering (or container preview),
- investigate the full import chain of
container.jsand related modules - mock or remove many Studio-only dependencies (api calls)
Disadvantages of this approach:
- Huge size of archive because we need to download all of the backbone libraries in any archive
- Dependencies must be carefully collected (big effort)
- Major changes in CMS (existing js files will have to be reused or supplemented)
I heard from Ed that Backbone.js will be obsolete in OpenEdx. frontend-app-authoring will use own render (not iframe with Backbone.js)
I heard from Ed that Backbone.js will be obsolete in OpenEdx. frontend-app-authoring will use own render (not iframe with Backbone.js)
This is partially correct, at most. While the vast majority of the legacy studio frontend is being removed, frontend-app-authoring will still iframe in the CMS backend's container_embed URL, which renders the new CMS templatecontainer-chromeless.html. The React-based editing UI is layered on top of the iframe. Here is the PR which added that template, if that helps: https://github.com/openedx/edx-platform/commit/e4a1e4136745ce02fc7424a3226f30d4c0965c8d
@kyrylo-kh Does that shed any light on a CMS-based solution? For example, would it be possible to use container-chromless.html as a guide for making a simplified component-chromeless.html, which renders an XBlock offline view with minimal unnecessary JS and CSS?
@kdmccormick
I investigated how XBlock is rendered in Open edX, what JS files are used, etc.
The conclusion: an XBlock can be rendered the same way in both LMS and CMS.
I built a small proof of concept.
To do this, you need to set up the XBlock correctly and provide the required services.
Here is a minimal working example:
services = {
"studio_user_permissions": StudioPermissionsService(request.user),
"i18n": XBlockI18nService,
"mako": mako_service,
"settings": SettingsService(),
"user": DjangoXBlockUserService(
request.user,
user_role=get_user_role(request.user, self.usage_key.course_key),
),
"partitions": StudioPartitionService(course_id=self.usage_key.course_key),
"teams_configuration": TeamsConfigurationService(),
"sandbox": SandboxService(contentstore=contentstore, course_id=self.usage_key.course_key),
"cache": CacheService(cache),
"replace_urls": ReplaceURLService,
}
Then you can call next code to get fragment:
block.render("student_view", context=student_view_context)
I also created a new HTML template called offline_container.html and included the minimal set of JS files needed to render the problem:
<script src="assets/common/js/vendor/jquery.js" type="text/javascript"></script>
<script src="assets/bundles/commons.5b0663a1e4e51afc5fb2.js" type="text/javascript"></script>
<script src="assets/bundles/ProblemBlockDisplay.0d6e10aabc21778c1aae.js" type="application/javascript"></script>
<script src="assets/bundles/XModuleShim.f78dfb4f07eae2a41a0c.js" type="application/javascript"></script>
<script src="assets/bundles/bootstrap.bundle.js" type="application/javascript"></script>
<script src="assets/common/js/xblock/core.js" type="text/javascript"></script>
<script src="assets/common/js/xblock/runtime.v1.js" type="text/javascript"></script>
<script src="assets/xmodule.js" type="text/javascript"></script>
With this setup it worked. In CMS, the archive is generated with these JS files and the problem renders successfully.
Next step: I’ll update the diagram and close this topic. From now on, generation will be handled on the CMS side.
@kdmccormick hi! Can we approve the ADR and move forward with the suggested solution?
