Passing open files (stdin) to Singularity different from Docker

[rupertnash]

Expected Behavior

CWL v1.1 conformance test 199 (stdin_shorcut) should work on both Docker and Singularity

#1378

Actual Behavior

Passes on Docker, with output:

     16     198    1111

Fails on Singularity with output:

  16  198 1111

This must is some difference in the passing of filehandles as the code in job._job_popen is basically identical.

NB: Running the commands in a shell in the singularity container can get the "correct" answer:

Singularity> wc < cwl-v1.1-main/tests/whale.txt 
  16  198 1111
Singularity> cat cwl-v1.1-main/tests/whale.txt | wc
     16     198    1111

Singularity> cat cwl-v1.1-main/tests/whale.txt | wc

For now, I'm disabling this test for singularity, but it should be enabled.

See also #1378

[rupertnash]

Note that when this is resolved, we can remove the exclusion from conformance-test.sh.

[mr-c]

I ran this test with Singularity v3.8.0 and it passed, so I think they fixed something on their side.

Workarounds for older versions of Singularity would be appreciated.

[GlassOfWhiskey]

The real problem here is that the width of each field returned by the wc command is not constant. In particular, the width is computed differently according to the file type: for regular files minimum width is 1, while for other types of files minimum width is 7 (https://github.com/coreutils/coreutils/blob/master/src/wc.c#L764)

Given that, the result strongly depends on how the stdin redirection is interpreted by the system. For instance, with the same Docker image:

docker run -i -v /path/to/whale.txt:/path/to/whale.txt sh -c "wc < /path/to/whale.txt"
16  198 1111
docker run -i -v /path/to/whale.txt:/path/to/whale.txt wc < /path/to/whale.txt
     16     198    1111

I think this command is not the best candidate to be used as a conformance test, as it is not so repeatable. Or at least, it should not be used as a test for stdin shortcut format in CWL files, because the format-related errors are not related with the actual support for such feature.

[mr-c]

@GlassOfWhiskey Good investigation! I just made https://github.com/common-workflow-language/common-workflow-language/issues/932 to track that.

A suggestion for a better stdin test is welcome for https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed