Support caches on NTFS mounts on Linux

[bastianeicher]

Background

Windows file systems such as NTFS do not store executable bits and do not allow non-Administators to create symlinks. The C# version of Zero Install handles this by recording the relevant files in .xbit and .symlink files alongside the .manifest file. Side note: In the long run I intend to remove the .xbit and .symlink files and simply use the equivalent data already recorded in the .manifest file when verifying an implementation's digest.

When sharing an implementation cache between a Windows and a Linux installation by mounting an NTFS volume in Linux this causes problems. The OCaml version of Zero Install does not read the .xbit and .symlink files and therefore generates mismatching manifest digests.

C# solution

When running the C# version of Zero Install on Linux it uses a heuristic to determine whether the file system holding the cache supports executable bits and symlinks. If it does not, Zero Install switches to the Windows-like handling described above.

This makes it possible to share caches across OSes. Everything works fine as long as:

• the NTFS mount on Linux is configured to mark all files as executable,
• none of the applications in the cache mind additional files being marked executable and
• none of the applications in the cache depend on symlinks being present.

I believe this is an acceptable compromise since it is defiantly better than cross-OS caches not working at all.

Suggested OCaml solution

I would suggest extended the OCaml version of Zero Install with similar functionality.

As a first step the manifest digest verification code could use the same heuristic to detect Windows mounts and ignore the file system's report of executable bits and symlinks and simply use the data from the .manifest file instead. The same heuristic could be used to reject any writes to caches on NTFS volumes for the time being.

As a next step the OCaml version could be extended to record executable bit and symlink data prior to extracting archives, like the C# version does. The write block on NTFS caches could then be removed again.

Summary

Rather than switching cache storage behavior based on the current OS Zero Install implementations should decide based on the underlying file system.

OS	File system	Storage of executable bits and symlinks
Windows	NTFS	Only in .manifest (and .xbit/.symlink)
Linux	ext3, etc.	In file system attribues
Linux	NTFS	Only in .manifest (and .xbit/.symlink)

This would result in any combination of Windows/Linux, C#/Ocaml 0install, NTFS/Unix-FS working.

[talex5]

Rather than using a heuristic, could we have a special file at the top-level of the whole cache to indicate this? e.g. /var/cache/0install.net/implementations/.no-xbits or something? Might be more reliable than guessing.

[bastianeicher]

I agree a flag file seems like a better solution. But who would be responsible for creating it? Any Zero Install instance running on Windows coming across an implementation directory that does not have one yet?

[talex5]

Yes, I think that makes sense (the check would be done on first write, I imagine).

[bastianeicher]

I'll add the corresponding code to the next Zero Install for Windows release. I'd go for .no-unix-fs if that's fine for you.

[talex5]

Sounds good to me!

[bastianeicher]

Ok, the .no-unix-fs generation code has been in the Windows releases for a while now, so most auto-updating users should have it by now.

Could you look into the OCaml side of this feature? I still have zero experience with the language.

[talex5]

Does this only affect 0install store audit / verify?

[bastianeicher]

It also affects 0install select / download / update / run since a user might set up 0install on Linux to write newly downloaded implementations to a cache stored on an NTFS partition.

[bastianeicher]

This could be split up into two steps:

1. Modify 0install store audit / verify to look for presence of a .no-unix-fs file. If this is detected take the "executable" and "symlink" states from the .manifest file rather than looking the file system itself. Additional value provided: Implementations downloaded on Windows can be verified on Linux.
2. Modify 0install store select / download / run to look for presence of a .no-unix-fs file. If this is detected write the "executable" and "symlink" states to the .manifest file but do not attempt to apply them to the file system. This would entail getting "executable" and "symlink" states from archives before extracting them. Additional value provided: Implementations downloaded on Linux can be used on Windows.