Upload collection of files error "could not store directory"

[zapaz]

Context

Bee version: Full node v1.17.4 System information: linux Environment: DappNode

Summary

Error 500 error could not store directory while uploading collection of files via Bee API

Expected behavior

Successful upload of all files, with reference returned

Actual behavior

When uploading collection of files via this API point https://docs.ethswarm.org/api/#tag/BZZ/paths/~1bzz/post got this 500 error could not store directory

Steps to reproduce

Launch this request (via curl or via js):

POST http://127.0.0.1:1633/bzz
Content-Type: multipart/form-data; boundary=YTJHJHG465HFG
Swarm-Postage-Batch-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

--YTJHJHG465HFG
Content-Disposition: form-data; name="swarm";
Content-Type: text/plain

BZZ
--YTJHJHG465HFG--

response with Error 500 :

HTTP/1.1 500 Internal Server Error
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
Date: Sun, 12 Nov 2023 10:21:33 GMT
Content-Length: 76
Connection: close

{
  "code": 500,
  "message": "could not store directory"
}

But this one is working with Content-Length: 3 (also work for whatever wrong length...)

POST http://127.0.0.1:1633/bzz
Content-Type: multipart/form-data; boundary=YTJHJHG465HFG
Swarm-Postage-Batch-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

--YTJHJHG465HFG
Content-Disposition: form-data; name="swarm";
Content-Type: text/plain
Content-Length: 3

BZZ
--YTJHJHG465HFG--

succesful response

HTTP/1.1 201 Created
Access-Control-Expose-Headers: Swarm-Tag
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Swarm-Tag: 180
Vary: Accept-Encoding
Date: Sun, 12 Nov 2023 10:35:15 GMT
Content-Length: 93
Connection: close

{
  "reference": "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
}

Possible solution

it seems that it's due to Content-Length required by Bee in each part of the multipart/form-data, although this behaviour is not compliant to RFC1867 and RFC2854

So API calls are not working with native browser fetch and FormData (moreover seems there exists no way to add Content-Length with FormData)

Possible solution : do not use non-compliant Content-Length for each part of a multipart or other solution: do not produce an error when Content-Length not present

[zapaz]

some context on why to solve this problem :

we would like to get multiple upload working the standard way (I mean with browser fetch and FormData). The tar method induces some problems : 1/ we loose 'Content-type' during the tar process, so the images are not directly diplayed (browser doesn't always know it's an image) on some Swarm gateways (may depend on what server gateway used) 2/ with the tar method, the hash of multiple part is not constant (not immutable), so that can be a problem if we have to re-apply same content 3/ It implies to write much more code and also depends on an external tar library, that doesn't gives same hashes with different langage or OS

Morover it seems , we are not alone to get this problem, and this induces very often this 'could not store directory' Error

[zapaz]

problem is here: https://github.com/ethersphere/bee/blob/master/pkg/api/dirs.go#L279-L281

remove this test, and set fileSize to 0 or calculate it with the real length of the part