docker-node icon indicating copy to clipboard operation
docker-node copied to clipboard
verified publisher icon nodejs

ci: add architectures update script

Open ttshivers opened this issue 5 years ago • 12 comments

This adds an action that checks whether the architectures files are outdated and will open a PR (TODO) with the changes. I saw this might be something potentially helpful when it was mentioned in #1312 by @nschonni .

At the moment, this grabs the available arches using bashbrew from https://github.com/docker-library/official-images for the base images. It then compares those arches to the ones stored in the */architectures file and modifies if they don't match.

TODO: Incorporate supported nodejs architectures

The script now just uses all the arches that the base image supports and nodejs might not support all of them. I haven't found a good programmatic way to get what arches nodejs supports. Some ways or ideas that I had include looking at the official dist release files, however it doesn't always have all the arches that this repo supports currently. I am interested in recommendations about the best way to do this.

Architectures file format

In this script, I used an object relating variant -> arches, while the architectures file stores data in a way that is equivalent to arch -> variants. I end up having to invert the structure to do comparisons. It seems more natural to store this data in a format like variant -> arches, since that's how it's also eventually translated to for the library files and how I get the data from bashbrew (it gives me a list of arches for a given image). (Potentially even store it as a json file for easy nodejs script integration)

ttshivers avatar Sep 30 '20 02:09 ttshivers 

diff --git a/14/architectures b/14/architectures
index b9eda44..9e87b80 100644
--- a/14/architectures
+++ b/14/architectures
@@ -1,8 +1,10 @@
-bashbrew-arch   variants
-amd64    stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm32v6  alpine3.10,alpine3.11,alpine3.12
-arm32v7  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm64v8  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-i386     alpine3.10,alpine3.11,alpine3.12
-ppc64le  buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-s390x    buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
+bashbrew-arch  variants
+arm32v6        alpine3.10,alpine3.11,alpine3.12
+mips64le       buster,buster-slim
+ppc64le        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim
+s390x          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim
+amd64          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
+arm32v7        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm64v8        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+i386           alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim

Think the arch value might need to be sorted too

nschonni avatar Sep 30 '20 02:09 nschonni

Might help to "cheat" and fix the indenting/alignment in the architecture files, so you get a cleaner diff in the same diffs

nschonni avatar Sep 30 '20 02:09 nschonni

Ah, I guess the initial one is going to create a diff no matter what since it also fixes the sorting of the image names.

diff --git a/14/architectures b/14/architectures
index b9eda44..bb656d5 100644
--- a/14/architectures
+++ b/14/architectures
@@ -1,8 +1,10 @@
-bashbrew-arch   variants
-amd64    stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm32v6  alpine3.10,alpine3.11,alpine3.12
-arm32v7  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm64v8  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-i386     alpine3.10,alpine3.11,alpine3.12
-ppc64le  buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-s390x    buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
+bashbrew-arch  variants
+amd64          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
+arm32v6        alpine3.10,alpine3.11,alpine3.12
+arm32v7        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm64v8        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+i386           alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+mips64le       buster,buster-slim
+ppc64le        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim
+s390x          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim

I guess there is a question on whether to include armv5 and mips64le

nschonni avatar Sep 30 '20 02:09 nschonni

Ah, I guess the initial one is going to create a diff no matter what since it also fixes the sorting of the image names.

diff --git a/14/architectures b/14/architectures
index b9eda44..bb656d5 100644
--- a/14/architectures
+++ b/14/architectures
@@ -1,8 +1,10 @@
-bashbrew-arch   variants
-amd64    stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm32v6  alpine3.10,alpine3.11,alpine3.12
-arm32v7  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-arm64v8  stretch,stretch-slim,buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-i386     alpine3.10,alpine3.11,alpine3.12
-ppc64le  buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
-s390x    buster,buster-slim,alpine3.10,alpine3.11,alpine3.12
+bashbrew-arch  variants
+amd64          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
+arm32v6        alpine3.10,alpine3.11,alpine3.12
+arm32v7        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm64v8        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+i386           alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+mips64le       buster,buster-slim
+ppc64le        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim
+s390x          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim

I guess there is a question on whether to include armv5 and mips64le

I'll align and sort the existing ones to make it easier

ttshivers avatar Sep 30 '20 02:09 ttshivers

Overall I like this. Probably need to figure out how to things might be filtered. EX: no support on 386 Linux or maybe armv5/mips

Yeah, that is the one thing that needs to be done. Either a manual allow list or block list or some way of handling that. Or possibly coding in a table or json file of supported nodejs version and arches and taking the intersection of those arches and the base image's arches.

ttshivers avatar Sep 30 '20 03:09 ttshivers

Just pulling out the diff

diff --git a/10/architectures b/10/architectures
index d31c0c2..6753515 100644
--- a/10/architectures
+++ b/10/architectures
@@ -1,8 +1,10 @@
 bashbrew-arch  variants
 amd64          alpine3.10,alpine3.11,alpine3.9,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
 arm32v6        alpine3.10,alpine3.11,alpine3.9
 arm32v7        alpine3.10,alpine3.11,alpine3.9,buster,buster-slim,stretch,stretch-slim
 arm64v8        alpine3.10,alpine3.11,alpine3.9,buster,buster-slim,stretch,stretch-slim
-i386           alpine3.10,alpine3.11,alpine3.9
+i386           alpine3.10,alpine3.11,alpine3.9,buster,buster-slim,stretch,stretch-slim
+mips64le       buster,buster-slim
 ppc64le        alpine3.10,alpine3.11,alpine3.9,buster,buster-slim
 s390x          alpine3.10,alpine3.11,alpine3.9,buster,buster-slim
diff --git a/12/architectures b/12/architectures
index 2d5e52a..a3170ba 100644
--- a/12/architectures
+++ b/12/architectures
@@ -1,8 +1,10 @@
 bashbrew-arch  variants
 amd64          alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
 arm32v6        alpine3.10,alpine3.11,alpine3.12,alpine3.9
 arm32v7        alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim,stretch,stretch-slim
 arm64v8        alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim,stretch,stretch-slim
-i386           alpine3.10,alpine3.11,alpine3.12,alpine3.9
+i386           alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim,stretch,stretch-slim
+mips64le       buster,buster-slim
 ppc64le        alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim
 s390x          alpine3.10,alpine3.11,alpine3.12,alpine3.9,buster,buster-slim
diff --git a/14/architectures b/14/architectures
index 09ace84..21eaa5e 100644
--- a/14/architectures
+++ b/14/architectures
@@ -1,8 +1,10 @@
 bashbrew-arch  variants
 amd64          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+arm32v5        buster,buster-slim,stretch,stretch-slim
 arm32v6        alpine3.10,alpine3.11,alpine3.12
 arm32v7        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
 arm64v8        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
-i386           alpine3.10,alpine3.11,alpine3.12
+i386           alpine3.10,alpine3.11,alpine3.12,buster,buster-slim,stretch,stretch-slim
+mips64le       buster,buster-slim
 ppc64le        alpine3.10,alpine3.11,alpine3.12,buster,buster-slim
 s390x          alpine3.10,alpine3.11,alpine3.12,buster,buster-slim

nschonni avatar Sep 30 '20 03:09 nschonni

One reference for supported platforms is: https://github.com/nodejs/node/blob/master/BUILDING.md#platform-list

ttshivers avatar Oct 01 '20 03:10 ttshivers

Using the Tier 1/Tier 2/Experimental on the multi-arch setup would be an interesting approach. Could mark the Tier 1 and maybe Tier 2 as required passing checks, but allow failures in the experimental

nschonni avatar Oct 03 '20 15:10 nschonni

Using the Tier 1/Tier 2/Experimental on the multi-arch setup would be an interesting approach. Could mark the Tier 1 and maybe Tier 2 as required passing checks, but allow failures in the experimental

Should those tiers play any role in how this action chooses variants or arches?

ttshivers avatar Oct 06 '20 22:10 ttshivers

Would probably need to go back through the issues in the build issue to get the exact platform reqs for the older releases, but here would be an examples for 14:

  • Tier 1:
    • Debian x64
    • Debian arm64
    • Debian armv7
  • Tier 2:
    • Debian s390x
    • Debian ppc64le
  • Experimental
    • Debian armv6
    • Debian x86
    • Alpine x64

nschonni avatar Oct 06 '20 22:10 nschonni

How should we store those tiers? In a json file somewhere or some other format?

ttshivers avatar Oct 06 '20 23:10 ttshivers

How should we store those tiers? In a json file somewhere or some other format?

It probably makes sense for the build group (or something) to store it in some machine readable format for us to consume. If we pull it into this repo it's bound to become outdated

SimenB avatar Oct 09 '20 09:10 SimenB