Consider using CoreOS developer container

[therc]

See https://gist.github.com/marineam/9914debc25c8d7dc458f It uses the same toolchain as the one that built the CoreOS kernel and it doesn't require running Ubuntu. I haven't tested it, but I think the current approach results in a much larger driver image than necessary.

Perhaps this approach would work, for a given CoreOS release XYZ:

1. Grab cuda_*run and extract it to a directory on the host machine. You can't do that inside the container, because it doesn't have enough space to download the archive, let alone expand it, etc.
2. Prepare a shell script to be run on the target nodes, which performs sanity checks, then runs insmod
3. Run container version XYZ using systemd-nspawn, adding --bind for the cuda directory, the script and any other files/directories needed. Make nspawn run a Makefile or similar (CoreOS doesn't ship with Make, but the container does!), which perhaps outputs a tar file with just the driver, a shell script and maybe insmod
4. Run docker import to create a Docker image from the tarball (which can be piped), with version XYZ and the shell script as the entrypoint
5. ???
6. Profit!

What do you think? There'd be no Dockerfile left, at the end, but it might be worth it.

[mikeorzel2]

Hi @therc

Agree, the containers are WAY too large at the moment. Will look into this and let you know what I see. The other thought I had was to stick with the current approach but just output the drivers for an insmod type container. It would mean the initial build would be large but the runtime container would be "just enough".

Thanks for the suggestion.

[therc]

The main reason to grab the container is to use the same compiler. I think the kernel refuses to load modules built with a different minor (4.8 vs 4.9). You're forcing the use of 4.9, so that works for now, but that might still result in subtle differences, if not bugs, when the compilers are not built with the same options and defaults.