simonihmig
How to convert to v2 addon?
Trying to investigate here how this addon could eventually be migrated to the v2 format.
Introduction
The special (unique?) property of this addon is that it has both a build-time part as well as a runtime. And both depend on each other, as the build generates the actual processed image files and provides some meta data about each image, that the runtime needs.
For this to work together, the addon uses these Ember CLI hooks, that are not available anymore for v2 addons:
-
contentFor -
treeForPublic -
treeForAddonStyles -
treeForFastBoot -
postBuild
Currently the addon fully works both in a classic build as well as under Embroider, albeit as a v1 addon using the compat layer.
How it works today
When the app builds, all images that match a configured pattern are processed by the addon. When there are n sizes configured, and m image types (jpg, png, webp, aviv), this will result in n x m variations for every image. These generated images are pushed (intentionally using this term) into the app, using the addon's treeForPublic hook. So the hook is not used for providing public files that the addon owns, but (processed) files owned by the app!
During the processing, meta data for each original image is collected (e.g. references to those n x m resulting images, each having their custom fingerprint). This meta data is essential for the runtime parts of the addon, so when <ResponsiveImage @src="path/to/original.jpg"/> is used, the component can look up the meta data and so the actual processed images based on path/to/original.jpg (as the key to a dictionary basically).
This meta data is injected into the index.html as a <script type="application/json">, using contentFor. Which later when the app has booted, the addon's service will query and decode.
Note: this meta data could have also been just added to
vendor.js. The motivation was to not invalidate the (cached and fingerprinted)vendor.jseach time something about the images changes. But not sure of this concern is really significant, as code is probably changing at a much higher frequency than (static!) images...
These are the most essential pieces. The use of the other hooks in a nutshell:
-
treeForAddonStyles: dynamically generated styles, based on the image processing step, for advanced "Low Quality Image Placeholder" (LQIP) features -
treeForFastBoot: The meta data is injected into a fastboot-only JS-bundle, as getting hold of the<script type="application/json">is not so easy in FastBoot, given the limitations ofSimpleDOM -
postBuild: remove original images from/dist(which could be huge, and are not needed when we have the processed ones)
Then there is an extra special case for the Blurhash LQIP feature. As browsers don't understand this format natively, a little JS script is needed to decode this data, into a PNG that the browser can understand. The addon creates a rollup build of the decode function and a little script, that converts the Blurhash data for all images in the page and applies it has a placeholder image, until the full image is loaded. In a SSR setup (FastBoot), it is important that this happens before the large bundles of the app are loaded and executed, otherwise the placeholder image would appear way too late for it to be useful. Therefore it is added, again using contentFor, to the index.html, but before any of the other app scripts.
How to migrate to v2
The addon could provide a webpack loader plugin, that, when a user imports an image, processes it and returns its meta data. That's a shift from push-based to pull-based, but given that's where we are heading to anyways, and also with explicit imports in strict mode, this could make a reasonable DX in the brave new world:
import ResponsiveImage from 'ember-responsive-image/components/responsive-image';
import myImage from `./path/to/my-image.jpg';
<template>
<ResponsiveImage @src={{myImage}}/>
</template>
In this case the loader would not return the path to the actual file, but the meta data that the component needs. This poses a possibly breaking change to the usual semantics of importing image assets, which is supposed to return the actual path after the build. Related: Asset Import Spec RFC. This could probably be worked around, by introducing a special hint which the loader will look for, like import myImage from './path/to/my-image.jpg?responsive';, or these webpack magic comments? 🤔
I think this should also just work in FastBoot, so no special handling required.
What about the treeForAddonStyles replacement? Can a loader, when importing an image, generate CSS, and later when all images have been processed add all that generated CSS to the other existing CSS? Not a webpack expert, but I doubt that!?
And what about the Blurhash support? The rollup build of the special script can be easily done at publish time of the v2 addon, as it does not need any of the app's images. But it needs to be added to the index.html, and somehow become part of the build output (i.e. it cannot stay in node_modules/ember-responsive-image/), and ideally be processed by the app's Babel settings.
Do we need to tell users they have to do some manual steps here instead, like
- create a
blurhash.jsscript somehwere in their app, that imports the actual code from the addon - add a
<script>reference to it in theirindex.html(before other scripts)
Would that already be enough, to make Embroider/webpack see that script as an additional entry point, and process it accordingly (Babel, bundling)?
Compatibility with classic builds
Assuming we got where we wanted to get with Embroider, so the magic is gone (for good or bad), and we need to tell users to import and add some things to their webpack config. Which obviously would be a breaking change, but that's ok. But would this setup then also work for a classic build w/ ember-auto-import, when applying the same webpack configuration?
I think most of what you need is already covered by the spec and current implementation.
Additional entrypoints
Embroider doesn't have a single privileged Javascript entrypoint. You're supposed to be able to introduce more just by using them from the HTML:
<script type="module" src="ember-responsive-image/blurhash"></script>
This should build a separate JS entrypoint starting from that file (which is resolved the normal node_modules resolution way), using the app's existing babel-present-env.
Styles and Public as Generated Imports
We already have a way to incorporate styles via imports that explicitly end in .css. So your transformation can emit styles into the build by emitting those imports.
Similarly, assuming something like the asset import spec you linked above (or assuming the app enables webpack's built-in asset handling that gives you URLs for imported assets), you can emit import statements for the image files you're building.
Consider an API where the user types this:
import { ResponsiveImage } from 'ember-responsive-image';
let Banner = ResponsiveImage("../images/banner.jpg");
<template>
<Banner />
</template>
And your transformation rewrites that to this:
import Image from 'ember-responsive-image/components/image'
import "../_generated/images/banner.jpg.css";
import jpg1x from "../_generated/images/banner.jpg.1x.jpg";
import jpg2x from "../_generated/images/banner.jpg.2x.jpg";
import webp1x from "../_generated/images/banner.jpg.1x.wepb";
import webp2x from "../_generated/images/banner.jpg.2x.webp";
let Banner = {
jpg1x,
jpg2x,
webp1x,
webp2x,
blurhash: "xxxxxxxx",
};
<template>
<Image @src={{Banner}} />
</template>
Notice that this can inject both styles and public assets, in a way that is legible to the final stage bundler. In particular, your plugin no longer needs to worry about things like production CDN prefixing of these images -- your code is just given a valid URL for each of the different images and doesn't need to concern itself with how that URL gets constructed.
There are two general strategies you might take to actually make the notional _generated directory that is illustrated above.
First, you may actually generate a real directory as a side-effect of your transformation. It could be in /tmp or a hidden directory so that it doesn't inconvenience users.
Second, you may instead emit URLs that trigger a further dedicated loader that will generate each of the files on demand, right in memory. For example, the webpack way of doing that might look like:
import Image from 'ember-responsive-image/components/image'
// the ember-responsive-image loader is responsible for emitting the CSS that goes with this image. The Embroider spec rules about incorporating that resulting CSS into the build will apply.
import "ember-responsive-image?type=css!../images/banner.jpg.css";
// similarly, the loader in these cases emits the actual image bits, and the runtime value that is returned to the code is a URL for them.
import jpg1x from "ember-responsive-image?size=1x&type=jpg!../images/banner.jpg";
import jpg2x from "ember-responsive-image?size=2&type=jpg!../images/banner.jpg";
import webp1x from "ember-responsive-image?size=1&type=webp!../images/banner.jpg";
import webp2x from "ember-responsive-image?size=2&type=webp!../images/banner.jpg";
let Banner = {
jpg1x,
jpg2x,
webp1x,
webp2x,
blurhash: "inline blurhash data goes here",
};
<template>
<Image @src={{Banner}} />
</template>
@ef4 thanks for the detailed response! I only now was able to look deeper into it...
I am not sure if I really understand you correctly though. As you are talking about a "transformation", and this before/after example above, you seem to suggest that this transform is transforming user code, is this what you are really suggesting?
My understanding was that the addon will ship one or multiple loaders, and these will determine what an import of an image will actually do and return. And I see a path towards this goal, your last example actually looks pretty much like what a loader could emit.
Taking my initial example: (with an inline specified addon-provided loader)
import ResponsiveImage from 'ember-responsive-image/components/responsive-image';
import myImage from `ember-responsive-image/loader!../images/banner.jpg';
<template>
<ResponsiveImage @src={{myImage}}/>
</template>
The loader could then return the following generated module: (the actual shape of the required meta data is very different, but this doesn't matter here)
import "ember-responsive-image/loader/css!../images/banner.jpg.css";
import jpg1x from "ember-responsive-image/loader/image?size=1x&type=jpg!../images/banner.jpg";
import jpg2x from "ember-responsive-image/loader/image?size=2&type=jpg!../images/banner.jpg";
import webp1x from "ember-responsive-image/loader/image?size=1&type=webp!../images/banner.jpg";
import webp2x from "ember-responsive-image/loader/image?size=2&type=webp!../images/banner.jpg";
import blurhash from "ember-responsive-image/loader/blurhash!../images/banner.jpg";
const meta = {
jpg1x,
jpg2x,
webp1x,
webp2x,
blurhash,
};
export default meta;
Does that makes sense? Again, this is almost the same as in your example, just showing how this could look like just with a custom loader. And no transform. But maybe I was just misunderstanding you...
But would this setup then also work for a classic build w/ ember-auto-import, when applying the same webpack configuration?
Could you also say something about this? Is this also going to work for ember-auto-import? And what about the additional entrypoint in index.html?
For the record: by coincidence I found this: https://github.com/dazuaz/responsive-loader
What is the status of the v2 branch?
I tried installing it on an application I am upgrading from Ember 3.x to Ember 4.12 with an Embroider build, but I run into this error when I execute ember server. The error is probably not a difficult one to fix, but I wonder whether it's the only one I would have to deal with.
npm ERR! [!] (plugin Typescript) SyntaxError: [...]/.npm/_cacache/tmp/git-cloneJ0NYKL/packages/ember-responsive-image/src/components/responsive-image.css: Unexpected token (1:0)
npm ERR!
npm ERR! > 1 | .eri-responsive {
npm ERR! | ^
npm ERR! 2 | width: 100%;
npm ERR! 3 | height: auto;
npm ERR! 4 | }
npm ERR! src/components/responsive-image.css (1:0)
npm ERR! SyntaxError: [...]/.npm/_cacache/tmp/git-cloneJ0NYKL/packages/ember-responsive-image/src/components/responsive-image.css: Unexpected token (1:0)
I also tried using ember-responsive-image @ 4.0, for which I created my own branch with updated dependencies, but it didn't work. ember server runs fine, but the metadata object filled in by the <script id="ember_responsive_image_meta"/> is empty. After some debugging, I noticed that the contentFor script in index.js is executed before images are built, thus, before the metadata object is filled.
I also tried using https://github.com/dazuaz/responsive-loader, but it does not pack the images in a structure sufficiently simple enough a usage as convenient as the original ember-responsive-image.
The v2 branch was working for me on Embroider last time I checked. But it's certainly not ready for general use. As stated in the PR checkboxes, the main missing piece is https://github.com/embroider-build/ember-auto-import/issues/565 for it to work with a classic build.
The current stable version should work with Embroider though. I might have to publish a new versions, since some renovate updates are probably not released yet. Why did you have to update dependencies in your branch, did something not work?
I tried the current stable version, but I run into this error on ember server
Build Error (PackagerRunner) in assets/app-name.js
Module not found: Error: Can't resolve './-private/function-based/modifier-manager' in '/path/to/app-repo/node_modules/.embroider/rewritten-packages/ember-modifier.8d972dc5'
Here is an extract of my package.json :
"@embroider/compat": "^3.1.5",
"@embroider/core": "^3.1.3",
"@embroider/webpack": "^3.1.3",
"ember-cli": "~4.12.1",
"ember-modifier": "^4.1.0",
Here is how the ember-responsive-image deps resolve:
├─┬ [email protected]
│ ├── @ember/[email protected] deduped
│ ├── @embroider/[email protected] deduped
│ ├── @glimmer/[email protected] deduped
│ ├── @glimmer/[email protected] deduped
│ ├─┬ @rollup/[email protected]
│ │ ├─┬ @rollup/[email protected]
│ │ │ ├── @types/[email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ └── @types/[email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ └── UNMET OPTIONAL DEPENDENCY fsevents@~2.3.2
│ ├─┬ @rollup/[email protected]
│ │ ├── @rollup/[email protected] deduped
│ │ ├─┬ @types/[email protected]
│ │ │ └── @types/[email protected]
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected] invalid: "^2.0.0" from node_modules/ember-responsive-image/node_modules/rollup-plugin-terser
│ │ ├── @types/[email protected] deduped
│ │ ├── @types/[email protected] deduped
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├─┬ @types/[email protected]
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── @types/[email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── @glimmer/[email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── @types/[email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ └─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── @babel/[email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ ├── @types/[email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ ├── [email protected] deduped invalid: "^2.0.0" from node_modules/ember-responsive-image/node_modules/rollup-plugin-terser
│ │ ├─┬ [email protected]
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected] overridden
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ ├─┬ [email protected]
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ └─┬ [email protected]
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├─┬ [email protected]
│ │ │ │ │ │ └── [email protected] deduped
│ │ │ │ │ └── [email protected] deduped
│ │ │ │ ├── [email protected] deduped
│ │ │ │ ├─┬ [email protected]
│ │ │ │ │ ├── [email protected]
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├── [email protected] deduped
│ │ │ │ │ ├─┬ [email protected]
│ │ │ │ │ │ ├── [email protected]
│ │ │ │ │ │ ├─┬ [email protected]
│ │ │ │ │ │ │ └── [email protected]
│ │ │ │ │ │ └── [email protected] deduped
│ │ │ │ │ ├─┬ [email protected]
│ │ │ │ │ │ └── [email protected]
│ │ │ │ │ └── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ ├─┬ [email protected]
│ │ │ │ │ └── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ ├── [email protected]
│ │ │ │ ├── [email protected] deduped
│ │ │ │ ├── [email protected]
│ │ │ │ └── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ [email protected]
│ │ │ └─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └── [email protected]
│ │ ├─┬ [email protected]
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │ ├── [email protected] deduped
│ │ │ ├─┬ [email protected]
│ │ │ │ └── [email protected]
│ │ │ └── [email protected] deduped
│ │ └─┬ [email protected]
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ └── [email protected] deduped
Note that I overrode the version of sharp because otherwise I get an undefined symbol error on my system with sharp >= 0.31 and Node (0.18.x). I considered using Node 0.16, I haven't tested it because I would like to avoid downgrading Node if possible.
If you think that the current stable shold work, perhaps I should just try to find the least amount of dependencies to update in order to fix each error I encounter?
Update: I fixed the above Module not found error with [email protected] by simply upgrading ember-style-modifier (through an overrides in my app's package.json).
However, I run into the empty metadata issue I mentioned earlier.
If I run ember build, whether for development and production, I can see the resized image files being generated, but the metadata remains empty.
<script id="ember_responsive_image_meta" type="application/json">
{"deviceWidths":[640,750,828,1080,1200,1920,2048,3840],"images":{}}
</script>