Allow known tools to be downloaded from alternate locations through some sort of config

[DamianAtWork]

We use Elm and downloading from GitHub releases directly is blocked. We do however have the facility to download binaries internally and host them at a location.

Would you consider allowing the location where the tools are downloaded to be configurable, so that folks like us who are behind restrictive corporate firewalls, can still access tools and versions we have explicitly downloaded and are serving up from an internal server?

[lydell]

Interesting! I’ll take a walk to think about this.

For inspiration, what are you currently using to download Elm (and elm-format etc)? / What does your current setup look like? I’m imagining you’re trying to replace something with the elm-tooling cli?

[DamianAtWork]

Currently we make them available internally on some file share and then we have some build scripts that properly setup the environment. It typically involves overriding the defaults and discovering where tools expect to find these tools.

BTW, one related, item is things like elm-test that use binwrap to download elmi-to-json don't work for us (because it reaches out to Github), and we've been forced to use (lobo)[https://www.npmjs.com/package/lobo] for testing because of this.

[lydell]

elm-tooling uses curl under the hood, so one way you might be able to do this already is to curl’s resolve option and a curl config file. For example:

❯ echo 'resolve = github.com:443:127.0.0.1' > .curlrc

❯ CURL_HOME=. curl https://github.com/foo
curl: (7) Failed to connect to github.com port 443: Connection refused

In this example I used 127.0.0.1 just to try things out, but maybe you can use the IP address of your server?

• curl config file: https://ec.haxx.se/cmdline/cmdline-configfile
• curl another host: https://daniel.haxx.se/blog/2018/04/05/curl-another-host/

Or maybe this can be used using proxies? https://ec.haxx.se/usingcurl/usingcurl-proxies

I’m not saying that I won’t add anything to elm-tooling for the use case, I’m just trying explore what’s already there first.

[DamianAtWork]

In this example I used 127.0.0.1 just to try things out, but maybe you can use the IP address of your server?

• curl config file: https://ec.haxx.se/cmdline/cmdline-configfile
• curl another host: https://daniel.haxx.se/blog/2018/04/05/curl-another-host/

Or maybe this can be used using proxies? https://ec.haxx.se/usingcurl/usingcurl-proxies

I can play with this a bit, but we don't really have an http endpoint to grab these from as far as I can tell (all this curl config stuff is new territory for me, though I can see how it could work).

I’m not saying that I won’t add anything to elm-tooling for the use case, I’m just trying explore what’s already there first.

Thanks for considering it. Having a first class workflow would really help us. Folks with internal artifactory servers and restrictions on using public repos (like us), really appreciation the ability to repoint to mirrors.

[lydell]

we don't really have an http endpoint to grab these from as far as I can tell

Interesting! So what kind of URL/scheme are they availble at then? Can you share an example URL?

[DamianAtWork]

we don't really have an http endpoint to grab these from as far as I can tell

Interesting! So what kind of URL/scheme are they availble at then? Can you share an example URL?

They are placed on a network share. The network share is replicated. So we use scripts which download them as such. I guess maybe I could try a file:/// type of thing with curl (once agin I'm not versed with curl 😄 ).

[lydell]

I played around with these curl things a little, and they are super fiddly because of HTTPS. It does not feel like a viable way to do this.

I’ll explore some kind of configuration next.

[lydell]

Here’s an idea: What if you could specify a JS file that exports a function that does the downloading in whatever way you want?

Maybe something like this (not sure about env var vs flag yet):

{
    "scripts": {
        "postinstall": "cross-env ELM_TOOLING_DOWNLOADER=./download.js elm-tooling install"
    }
}

download.js:

import * as fs from "fs";
import * as path from "path";

export function download({
  url,
  onData,
  onProgress,
  onError,
  onSuccess,
}: {
  name: string;
  version: string;
  url: URL;
  os: "linux" | "mac" | "windows";
  onData: (buffer: Buffer) => void;
  onProgress: (percentage: number | string) => void;
  onError: (error: Error) => void;
  onSuccess: () => void;
}): { kill: () => void } {
  const file = path.join("E:", "github-releases", url.pathname);

  let stat: fs.Stats;
  try {
    stat = fs.statSync(file);
  } catch (error) {
    onError(error);
  }
  const reader = fs.createReadStream(file);

  reader.on("error", onError);
  reader.on("close", onSuccess);

  let length = 0;
  reader.on("data", (chunk: Buffer) => {
    length += chunk.length;
    onData(chunk);
    onProgress(length / stat.size);
  });

  return {
    kill: () => {
      reader.close();
    },
  };
}

I just realized the above is TypeScript, but it would have to be regular JavaScript.

In CI you could run elm-tooling install without the env var/flag.

[DamianAtWork]

I like the idea. What we are looking for is pluggable flexibility and this would certainly cover that.

[lydell]

I have another idea cooking. Could you do me a favor and try one little thing? Run this in a Node.js repl:

fs.readFileSync("path-to-elm-binary-on-your-network-share").length

Is it able to read the file? If it is – great! If not, my new idea needs mor thinking.

[DamianAtWork]

So when I ran this I got the following:

let elmPath = "\netshare\dist\fsf\PROJ\elm-compiler\0.19.1\common\elm.exe"
Thrown:
fs.readFileSync(elmPath).length
Thrown:
TypeError [ERR_INVALID_ARG_VALUE]: The argument 'path' must be a string or Uint8Array without null bytes. Received '\netshare\fsfPROJelm-compiler\u0000.19.1commonelm.exe' at Object.openSync (fs.js:435:3)
at Object.readFileSync (fs.js:343:35)

[lydell]

Looks like you forgot to double up all backslashes?

let elmPath = "\\netshare\\dist\\fsf\\PROJ\\elm-compiler\\0.19.1\\common\\elm.exe"

[DamianAtWork]

Looks like you forgot to double up all backslashes?

let elmPath = "\\netshare\\dist\\fsf\\PROJ\\elm-compiler\\0.19.1\\common\\elm.exe"

You were right, that fixed it. And I get back the number as expected.

[lydell]

I just wanted to let you know I haven’t forgotten about this. It’s just that I don’t need this myself at the moment and I’ve been focusing on other exciting Elm projects (such as https://github.com/lydell/elm-safe-virtual-dom/).

Here’s the plan (iterated on Discord):

{
  "tools": {
    "elm": "0.19.1",
    "elm-test-rs": {
      "x86_64-linux": {
        "url": "https://github.com/mpizenberg/elm-test-rs/releases/download/v1.0.0-alpha/elm-test-rs_linux.tar.gz",
        "hash": "sha256-398feadd82ff3bce66dd953ff0093e27ac91a5c3981fc7d6048ff12451c0e774"
      },
      "aarch64-darwin": {
        "url": "https://github.com/mpizenberg/elm-test-rs/releases/download/v1.0.0-alpha/elm-test-rs_macos.tar.gz",
        "hash": "sha256-8ff62d759a60d1a0499bd5416f86c74a58b0635429892ef8a7b65af571762c5e"
      },
      "x86_64-darwin": {
        "url": "https://github.com/mpizenberg/elm-test-rs/releases/download/v1.0.0-alpha/elm-test-rs_macos.tar.gz",
        "hash": "sha256-8ff62d759a60d1a0499bd5416f86c74a58b0635429892ef8a7b65af571762c5e"
      },
      "x86_64-windows": {
        "url": "https://github.com/mpizenberg/elm-test-rs/releases/download/v1.0.0-alpha/elm-test-rs_windows.zip",
        "hash": "sha256-850ba50b633453b745279e8df31341df4395e754ce3ad25b776ff9fbfb0b863e"
      }
    }
  }
}

• You can use version numbers for some tools, and url+hash pairs for others. Mix as you please.
• URLs have to start with https:// or file://.
• URLs have to end with .tgz, .tar.gz, .gz or .zip (Linux does not support .zip for the time being).
• .tgz, .tar.gz and .zip are expected to contain a tool-name or tool-name.exe file, which is already chmod +x:ed.
• You can list 1–4 platforms. Open source projects are encouraged to list as many as they can, to help new contributors.
• Only these platforms are supported:
  • x86_64-linux
  • aarch64-darwin
  • x86_64-darwin
  • x86_64-windows
• It’s implementation specific how to match the user’s system to one of the above platforms.
• The hash is used in place of the version in the file path. For example, ~/.elm/elm-tooling/elm-test-rs/sha256-398feadd82ff3bce66dd953ff0093e27ac91a5c3981fc7d6048ff12451c0e774/elm-test-rs.

Stuff I want to do:

• Add the above format to the spec.
• Support the above format.
• Rewrite known-tools.ts to use that format too. The version numbers will just be aliases for objects of url+hashes.
• Increase test coverage by:
  • Having tests with invalid hashes.
  • Having tests that go to a local server that fails in given ways.
• Add a CLI command that interactively generates the JSON needed for custom urls. It will calculate the hashes for you.

Just to re-iterate the use cases:

1. Supporting environments where GitHub Releases is blocked
2. Allowing people to use new tools, such as elm-test-rs, before they are built-in supported.
3. In case people who have permission to merge PRs in elm-tooling/elm-tooling-cli are slow at merging support for a new tool version for some reason, and someone desperately wants the new version, they could do it at any time (with just a little more work on their own).

Edit: Important: Keep https://github.com/elm-tooling/elm-tooling-cli/issues/88 in mind when designing the JSON format. We might want to support something like (but specifying both node and some arch should not be allowed – or maybe it should, it could have the lowest precedence):

{
  "tools": {
    "elm-test": {
      "node": {
        "url": "https://registry.npmjs.org/elm-test/-/elm-test-0.19.1-revision6.tgz",
        "hash": "sha512-4VbIyCRlCUm/py0E0AjMT3/mwd6DR4Y5Z5gEox6z5JII6ZdKIJmcQzjgWRI5qo5ERJiw9M/Nxhk7SGXFUbZsxQ=="
      }
    }
  }
}

[DamianAtWork]

Oh man, this would be so amazing. Thanks so much.

[lydell]

It’s been a long time without updates! Here is the status: https://github.com/elm-tooling/elm-tooling-cli/issues/95