In production, images are not served for translated-content (whereas it works locally)

[SphinxKnight]

Summary

The images used in live sample ({{EmbedLiveSample}}) are not served (404) when they are not duplicated on mdn/translated-content.

URL

https://developer.mozilla.org/fr/docs/Web/CSS/filter

Reproduction steps

1. Go to https://developer.mozilla.org/fr/docs/Web/CSS/filter
2. Checkout mdn/translated-content as of e1b9e2c and a local yari
3. Compare live samples in the page

Expected behavior

Both (local and production) should match (at least after some time)

Actual behavior

In production https://yari-demos.prod.mdn.mozit.cloud/fr/docs/Web/CSS/filter/test_form_3.jpeg is 404

Device

Laptop

Browser

Firefox

Browser version

Stable

Operating system

Windows

Screenshot

Anything else?

Previously https://github.com/mdn/yari/pull/5215 was merged, it worked locally, I waited a bit and made https://github.com/mdn/translated-content/pull/4549

https://github.com/mdn/translated-content/issues/3752

Validations

• [X] I have read the Community Participation Guidelines.
• [X] I have verified that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] I have checked that this is a concrete bug. For Q&A open a GitHub Discussion.

[yin1999]

I have an idea about this problem, is it easier to build with correct image path ? It means that if the image is not in relative path at translated-content repo but in the content repo, we just replace the relative path with the absolute path (to content repo) before writing to index.html file.

Edit: seems a bit difficult, for we have a lot of ways to reference a image (CSS, javascript, HTML, SVG, etc..)

This doesn't complicate the proxy any more (proxy is needed to check whether the image file exists in en-US when S3 report that the file is not exists in l10n).

My english is not very well, apologies.

[SphinxKnight]

@mdn/core-yari-dev is it possible to have a look at this one? (ex. https://developer.mozilla.org/fr/docs/Web/CSS/filter still failing at loading resources)

[yin1999]

I found two function which may help with this.

• checkImageReferences in build\check-images.js: is used to fallback the src attribute of img to en-US sources while the src does not exist in l10n.
• buildLiveSamplePages in kumascript\src\live-sample.js: is used to build _sample_.some_id.html, but it will not check the image reference

Note that, livesample in https://developer.mozilla.org/fr/docs/Web/CSS/filter does have element image with attribute xlink:href.

[SphinxKnight]

@mdn/core-dev should I revert https://github.com/mdn/yari/pull/5215? is there anything we can do here?

[caugner]

@SphinxKnight We believe that #5215 is not the cause. It just doesn't seem to work for live samples (yet), and so removing the corresponding images from the translated-content repository caused the images to no longer show.

For now, the best solution would be to restore those images in translated-content.

[yin1999]

I have a new idea about resolving this problem.

Could we just generate a json file to list paths of files in mdn/content (exclude .md and .html file). just like #6124.

So we can use aws-lambda to redirect (302 redirect) the request of image file to en-US by modifying the response. Examples on aws.

The condition we may use:

• status code is 404 (file not exists in l10n)
• the requested file name (exclude .md and .html) exists in the JSON file
• the URL path has prefix like zh-CN, zh-TW, fr, etc (exclude en-US).

And I found a answer about image redirection on stackoverflow.

[schalkneethling]

Friendly ping on this one. @SphinxKnight, @yin1999, @caugner should we open a discussion to explore options here?

[yin1999]

should we open a discussion to explore options here?

Agree, we should gather more ideas.

[yin1999]

I've created #6644 to fix this. But i'm not really sure whether it's a nice solution.

[yin1999]

A new idea about this:

We are now hosting about 9 locales. And we may modify the request before finding the file hosted on aws-S3 (so we can reduce redirects for those images which have to fallback to en-US, the method metioned above)

So, we may go on this way (similar to the redirect logic of lambda@edge):

1. collect all those image paths in mdn/content repo
2. for each image path, check whether it's in l10n folders. If not, flag it.
3. store those flags of image path in a file
4. in Origin request, check whether the request is to l10n folder. If so, and the filepath has been flaged, modify the request.uri to en-us folder.

The problem is, there are too many images that are not in most l10n folders:

for about 1800 non-md files, we should create about 18000 flags (about 10 times) for those files are not in l10n.

So this may not be a good idea to store each not-existed l10n image file path. (if there are 4000 image files in mdn/content in the future, we should store about 40000 filepaths).

Another way to flag those file is bitwise flag (just like the Unix Octal permissions):

For each locale, we use a powers of 2 number to present. For example:

locale	number	bitwise
en-us	1	0000001
zh-cn	2	0000010
zh-tw	4	0000100
fr	8	0001000

If path/to/image does not exist in fr and zh-cn (but in other locales), we can compress the flags by sum up all the numbers: 2 (for zh-cn) + 8 (for fr) = 10.

In Origin request, we can:

• Check whether the path/to/image exists in fr: 10 (for sumed flags) & 8 (for fr) = 1. The result is not zero, so the image is not in fr.
• Check whether the path/to/image exists in zh-tw: 10 (for sumed flags) & 4 (for zh-tw) = 0. The result is zero, so the image is in zh-tw.
• ...

For the sumed flags, we can just use a map to present (easy to stored as a json file):

{
  “path/to/image_1”: 10,
  "path/to/image_2": 8,
  // etc.
}

And in JavaScript, the bitwise operation is safe for 32-bit integer, so we can host 32 locales at most. I think it's enough for us for a long time (when we host more than 32 locales, we may have a new design about image file hosting)

Filled in #6644.