s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

incorrect results for keys ending in `/` (eg as created by s3 console)

Open timotheecour4 opened this issue 3 years ago • 0 comments

bugs with s5cmd ls

echo hello > /tmp/z00.txt
aws s3api put-object --bucket mybucket --key bar/ --body /tmp/z00.txt

# bug 1: shows DIR instead of an object when specifying full path:
s5cmd ls s3://mybucket/bar/
                                  DIR bar/

# ok:
s5cmd ls s3://mybucket/bar/bar/
ERROR "ls s3://mybucket/bar/bar/": no object found

# bug 2: shows DIR instead of an object when specifying a prefix path:
s5cmd ls 's3://mybucket/bar*'
                                  DIR bar/

# bug 3: same as bug2
s5cmd ls 's3://mybucket/ba'
                                  DIR bar/

# bug 4 (even stranger): shows full path
s5cmd ls 's3://mybucket/bar/*'
                                  DIR s3://mybucket/bar/

In contrast, aws s3 also has bug 2 but not bug 1:

# ok (unlike bug 1, it correctly shows the object size etc instead of PRE)
aws s3 ls s3://mybucket/bar/
2022-09-23 10:27:39          6

# bug 6 (same as bug 2)
aws s3 ls 's3://mybucket/bar'
                           PRE bar/

bugs with s5cmd cp

these are even more troubleseome:

aws s3api put-object --bucket ticour-experiments2 --key a/bar/ --body /tmp/z00.txt
aws s3api put-object --bucket ticour-experiments2 --key a/baz --body /tmp/z00.txt

# bug: s5cmd copies a single key:
s5cmd cp 's3://ticour-experiments2/a/*' 's3://ticour-experiments2/a3/'
cp s3://ticour-experiments2/a/baz s3://ticour-experiments2/a3/baz

# ok: aws cp copies both keys:
aws s3 cp --recursive 's3://ticour-experiments2/a/' 's3://ticour-experiments2/a2/'
copy: s3://ticour-experiments2/a/baz to s3://ticour-experiments2/a2/baz
copy: s3://ticour-experiments2/a/bar/ to s3://ticour-experiments2/a2/bar/

note: where such keys come from in practice

after investigation in the root cause of what created such keys ending in /, it turns out it was created via s3 console (https://s3.console.aws.amazon.com/) which allows one to "Create folder" (which ends up creating a key ending in /). So this use case seems to be worth supporting given that s3 console has an explicit action to create those.

timotheecour4 avatar Sep 23 '22 17:09 timotheecour4