maint(common): fix running docker on Windows

[ermshiperete]

By default Git Bash converts paths starting with /, so running the docker image fails with errors or files mentioning \Program Files\Git\ (e.g. https://github.com/keymanapp/keyman/pull/14058#issuecomment-2951820833). This change adds a wrapper function that prevents the conversion and thus uses the paths in the correct form when calling docker.

Since the docker builds run on Linux they expect LF line endings. If the scripts are checked out with CRLF line endings, they look for things like bash^M. While there are workarounds to some degree, I didn't found a solution that would fully work with our scripts. So in the end this PR now always checks out scripts with LF line endings, even on Windows.

If you test this you'll have to re-checkout the source tree so that you end up with LF line endings.

Fixes: #14153 Test-bot: skip

[keymanapp-test-bot[bot]]

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

• Android

• Developer

  • Keyman Developer - build pending
  • Compiler Regression Tests - build pending
  • kmcomp.zip - build pending

• iOS

  • TestFlight internal PR build version - 19.0.101 (0.14154.14293)

• Keyboards

  • Test Keyboards - build pending

• macOS

• Web

• Windows

[ermshiperete]

https://stackoverflow.com/a/14610574 has a workaround, but that doesn't fully work for our scripts because then things fail with

/dev/fd/63: line 7: /proc/576/fd/resources/build/builder.inc.sh: No such file or directory
/dev/fd/63: line 10: builder_describe: command not found
/dev/fd/63: line 28: builder_describe_platform: command not found
/dev/fd/63: line 34: builder_parse: command not found
/dev/fd/63: line 38: builder_run_child_actions: command not found

because we convert the script on the fly and then read it from stdin. Even if we solve that problem I guess sourcing the include scripts still wouldn't work because they still have CRLF line endings.

See also https://blog.programster.org/fixing-docker-volume-windows-line-endings-on-bash-scripts.

[ermshiperete]

I hope it is acceptable to always checkout scripts (*.sh) with LF line endings, even on Windows.

[ermshiperete]

@mcdurdin Does this work for you?

On my Windows machine, when I try to build I get failures:

Details

[common/web/keyman-version] ## configure starting...
Now using node v20.16.0 (npm v10.8.1)
npm warn deprecated @humanwhocodes/[email protected]: Use @eslint/config-array instead
npm warn deprecated @humanwhocodes/[email protected]: Use @eslint/object-schema instead
npm warn cleanup Failed to remove some directories [
npm warn cleanup   [
npm warn cleanup     '/home/build/build/node_modules/fill-range',
npm warn cleanup     [Error: EACCES: permission denied, rmdir '/home/build/build/node_modules/fill-range'] {
npm warn cleanup       errno: -13,
npm warn cleanup       code: 'EACCES',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/home/build/build/node_modules/fill-range'
npm warn cleanup     }
npm warn cleanup   ],
npm warn cleanup   [
npm warn cleanup     '/home/build/build/node_modules',
npm warn cleanup     [Error: EACCES: permission denied, rmdir '/home/build/build/node_modules/has-property-descriptors'] {
npm warn cleanup       errno: -13,
npm warn cleanup       code: 'EACCES',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/home/build/build/node_modules/has-property-descriptors'
npm warn cleanup     }
npm warn cleanup   ],
npm warn cleanup   [
npm warn cleanup     '/home/build/build/node_modules/portfinder',
npm warn cleanup     [Error: EACCES: permission denied, rmdir '/home/build/build/node_modules/portfinder/node_modules/debug'] {
npm warn cleanup       errno: -13,
npm warn cleanup       code: 'EACCES',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/home/build/build/node_modules/portfinder/node_modules/debug'
npm warn cleanup     }
npm warn cleanup   ],
npm warn cleanup   [
npm warn cleanup     '/home/build/build/node_modules/tsx',
npm warn cleanup     [Error: EACCES: permission denied, rmdir '/home/build/build/node_modules/tsx/dist'] {
npm warn cleanup       errno: -13,
npm warn cleanup       code: 'EACCES',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/home/build/build/node_modules/tsx/dist'
npm warn cleanup     }
npm warn cleanup   ],
npm warn cleanup   [
npm warn cleanup     '/home/build/build/node_modules/@microsoft',
npm warn cleanup     [Error: EACCES: permission denied, rmdir '/home/build/build/node_modules/@microsoft/tsdoc-config/node_modules/ajv/lib/compile'] {
npm warn cleanup       errno: -13,
npm warn cleanup       code: 'EACCES',
npm warn cleanup       syscall: 'rmdir',
npm warn cleanup       path: '/home/build/build/node_modules/@microsoft/tsdoc-config/node_modules/ajv/lib/compile'
npm warn cleanup     }
npm warn cleanup   ]
npm warn cleanup ]
npm error code EPERM
npm error syscall chmod
npm error path /home/build/build/node_modules/@keymanapp/resources-gosh/gosh.js
npm error errno -1
npm error Error: EPERM: operation not permitted, chmod '/home/build/build/node_modules/@keymanapp/resources-gosh/gosh.js'
npm error     at async chmod (node:internal/fs/promises:1085:10)
npm error     at async Promise.all (index 0)
npm error     at async Promise.all (index 0)
npm error     at async #createBinLinks (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/rebuild.js:394:5)
npm error     at async Promise.allSettled (index 2)
npm error     at async #linkAllBins (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/rebuild.js:375:5)
npm error     at async #build (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/rebuild.js:160:7)
npm error     at async Arborist.rebuild (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/rebuild.js:67:7)
npm error     at async [reifyPackages] (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:325:11)
npm error     at async Arborist.reify (/home/build/.nvm/versions/node/v20.16.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:142:5) {
npm error   errno: -1,
npm error   code: 'EPERM',
npm error   syscall: 'chmod',
npm error   path: '/home/build/build/node_modules/@keymanapp/resources-gosh/gosh.js'
npm error }
npm error
npm error The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user
npm error
npm error If you believe this might be a permissions issue, please double-check the
npm error permissions of the file and its containing directories, or try running
npm error the command again as root/Administrator.
npm error A complete log of this run can be found in: /home/build/.npm/_logs/2025-06-12T16_42_52_743Z-debug-0.log

[mcdurdin]

@ermshiperete Yeah, this doesn't work for me either. First, I had to fix my line endings. I don't disagree with this setting, and I wonder if we should be extending it to other file types as well.

Line endings kerfuffle first

The following command apparently will refresh line endings, for all files.

git add --renormalize .

There are a number of unrelated files touched, which I found interesting but not relevant. So I rolled those back. We could restrict the renormalize match to *.sh:

shopt -s extglob nullglob globstar
git add --renormalize **/*.sh

But --renormalize didn't work for me -- line endings stayed as CRLF for .sh (cat -A is an easy way to check line endings). I checked core.autocrlf, and it was set to true so turned it off. But --renormalize still didn't refresh line endings on .sh files.

I am probably doing something wrong, but gave up on that and just did this (to avoid munging untracked files), which worked:

find . -name '*.sh' | xargs -l git rm
git checkout .

Trying with 'right' line endings

This got line endings resolved. I am now in the same situation as you, Eberhard.

I ran into the same kinds of issues with npm as you did, and realized it's because npm on Linux is assuming Unix-style symlinks whereas on Windows these are implemented differently. So I removed all the node_modules folders, and tried again.

find . -type d -name 'node_modules' | xargs rm -rf

But still ran into permissions problems. I am not sure how easy this is to resolve.

winpty side-track

If running in git bash (as opposed to a VSCode bash term), I found I needed to prefix docker calls with winpty to avoid getting the error the input device is not a TTY. If you are using mintty, try prefixing the command with 'winpty':

    MSYS_NO_PATHCONV=1 winpty docker "$@"

But adding winpty broke volume mapping. I was able to resolve that with a tip found at https://pitman.io/posts/tips-for-using-docker-from-git-bash-on-windows/ (not sure if this would break other platforms though), adding a / prefix to each of the -v commands:

  docker_wrapper run -it --rm -v /"${KEYMAN_ROOT}":/home/build/build \
    -v /"${KEYMAN_ROOT}/core/build/docker-core/${build_dir}":/home/build/build/core/build \
    "keymanapp/keyman-developer-ci:${image_version}" \
    "${builder_extra_params[@]}"

But this doesn't fix the symlink issue. And winpty breaks pipes so this should really only be used with -it docker runs.

[ermshiperete]

It does seem to be a permission problem:

In the Docker image we create a build user that has the same uid/gid as the first user on an Ubuntu system (1000/1000), and then use this user when building inside the docker container. If we run this docker container on a Linux host, files we create in the container will have the same permissions and owner as if we'd run the build in the host, because the uids in the container and in the host match. Similarly, root (uid 0) in the container and in the host match.

However, on a Windows host when we start Docker from git bash this works differently: git bash/mingw does a user mapping from the Windows user to the mingw environment. On my machine this results in uid/gid 197610. When we start Docker from mingw it seems to map this uid to the root user (uid 0) in the container. The build user we create therefore doesn't match the user on the Windows system nor on mingw. If we create a file in the container as build user, it shows up as being owned by our Windows user. However, the same is true if we create the file as root in the container. (I didn't check the Windows owner/permissions though, but ls -al in git bash shows the file as being owned by my Windows user). The bind-mounted Keyman source directory inside the container shows up as being owned by root, so the error message is correct.

What works is to chown -R build:build /home/build/build inside the container, but that takes time. It doesn't seem to change anything on the Windows side, but makes things work inside the container.

Another solution which I finally implemented is to run the build as root user when we start the docker container from mingw/git bash.

[ermshiperete]

With the latest commit building developer in docker works for me on Windows if I start with a clean repo. Things don't work if I mix building on Docker and on Windows, e.g. I run configure from Windows and then try to do a build in docker. The reason are different names for the tools (foo.exe vs foo). The solution will be to add another bind mount for building developer in docker, similar to what we do for Core and Linux.

[ermshiperete]

The solution will be to add another bind mount for building developer in docker, similar to what we do for Core and Linux.

The only problem with this is that we have a lot of small subprojects with a build folder in each subproject. Maybe we could use overlayfs to solve this?

[ermshiperete]

overlayfs might work and be an elegant solution, but it's not available in git bash, so doesn't really help. I'm not sure if any copy-on-write system is available on Windows that works on the file/directory level - a quick search didn't show anything.

[mcdurdin]

Sharing a repo between Linux and Windows is always going to be fraught - node_modules for example is very platform-specific in what is installed in it. Let's not try and make it work both ways on a single copy of the repo: the developer needs to decide if they will run everything in WSL/Docker or run everything in Windows/git-bash.

[keyman-server]

Changes in this pull request will be available for download in Keyman version 19.0.103-alpha