s5cmd
s5cmd copied to clipboard
incorrect results for keys ending in `/` (eg as created by s3 console)
bugs with s5cmd ls
echo hello > /tmp/z00.txt
aws s3api put-object --bucket mybucket --key bar/ --body /tmp/z00.txt
# bug 1: shows DIR instead of an object when specifying full path:
s5cmd ls s3://mybucket/bar/
DIR bar/
# ok:
s5cmd ls s3://mybucket/bar/bar/
ERROR "ls s3://mybucket/bar/bar/": no object found
# bug 2: shows DIR instead of an object when specifying a prefix path:
s5cmd ls 's3://mybucket/bar*'
DIR bar/
# bug 3: same as bug2
s5cmd ls 's3://mybucket/ba'
DIR bar/
# bug 4 (even stranger): shows full path
s5cmd ls 's3://mybucket/bar/*'
DIR s3://mybucket/bar/
In contrast, aws s3 also has bug 2 but not bug 1:
# ok (unlike bug 1, it correctly shows the object size etc instead of PRE)
aws s3 ls s3://mybucket/bar/
2022-09-23 10:27:39 6
# bug 6 (same as bug 2)
aws s3 ls 's3://mybucket/bar'
PRE bar/
bugs with s5cmd cp
these are even more troubleseome:
aws s3api put-object --bucket ticour-experiments2 --key a/bar/ --body /tmp/z00.txt
aws s3api put-object --bucket ticour-experiments2 --key a/baz --body /tmp/z00.txt
# bug: s5cmd copies a single key:
s5cmd cp 's3://ticour-experiments2/a/*' 's3://ticour-experiments2/a3/'
cp s3://ticour-experiments2/a/baz s3://ticour-experiments2/a3/baz
# ok: aws cp copies both keys:
aws s3 cp --recursive 's3://ticour-experiments2/a/' 's3://ticour-experiments2/a2/'
copy: s3://ticour-experiments2/a/baz to s3://ticour-experiments2/a2/baz
copy: s3://ticour-experiments2/a/bar/ to s3://ticour-experiments2/a2/bar/
note: where such keys come from in practice
after investigation in the root cause of what created such keys ending in /, it turns out it was created via s3 console (https://s3.console.aws.amazon.com/) which allows one to "Create folder" (which ends up creating a key ending in /). So this use case seems to be worth supporting given that s3 console has an explicit action to create those.