Object Upload fails but still set as accepted on chain

[yasiryagi]

Issue:

Uploads that has failed , but still marked as accepted on chain

Evidence:

dataObjectId: 2100687

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "2100687"
      }
    ]
  }
}

Internal Server Error: Upload aborted
    at error (/joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:123:20)
    at /joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:16:26
    at Immediate._onImmediate (/joystream/node_modules/multer/lib/make-middleware.js:53:37)
    at process.processImmediate (node:internal/timers:478:21)

dataObjectId: 2112736

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "2112736"
      }
    ]
  }
}

uncaughtException: Upload aborted
Internal Server Error: Upload aborted
    at error (/joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:123:20)
    at /joystream/node_modules/express-openapi-validator/dist/middlewares/openapi.multipart.js:16:26
    at Immediate._onImmediate (/joystream/node_modules/multer/lib/make-middleware.js:53:37)
    at process.processImmediate (node:internal/timers:478:21)

Error: Unexpected end of JSON input
    at new ApolloError (/joystream/node_modules/@apollo/client/errors/errors.cjs:34:28)
    at /joystream/node_modules/@apollo/client/core/core.cjs:1654:19
    at both (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:986:53)
    at /joystream/node_modules/@apollo/client/utilities/utilities.cjs:979:72
    at new Promise (<anonymous>)
    at Object.then (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:979:24)
    at Object.error (/joystream/node_modules/@apollo/client/utilities/utilities.cjs:987:49)
    at notifySubscription (/joystream/node_modules/zen-observable/lib/Observable.js:140:18)
    at onNotify (/joystream/node_modules/zen-observable/lib/Observable.js:179:3)
    at SubscriptionObserver.error (/joystream/node_modules/zen-observable/lib/Observable.js:240:7)

dataObjectId: 1713397

{
  "data": {
    "storageDataObjects": [
      {
        "isAccepted": true,
        "id": "1713397"
      }
    ]
  }
}

uncaughtException: Timeout
Error
    at new module.exports.TimeoutError (/joystream/node_modules/promise-timeout/index.js:44:16)
    at module.exports.timeout (/joystream/node_modules/promise-timeout/index.js:20:15)
    at extrinsicWrapper (/joystream/storage-node/lib/services/runtime/extrinsics.js:207:45)
    at acceptPendingDataObjects (/joystream/storage-node/lib/services/runtime/extrinsics.js:115:18)
    at uploadFile (/joystream/storage-node/lib/services/webApi/controllers/filesApi.js:99:57)
```	
dataObjectId: 2112361

{ "data": { "storageDataObjects": [ { "isAccepted": true, "id": "2112361" } ] } }

Error: Server is not accepting uploads into this bucket at validateUploadFileParams (/joystream/storage-node/lib/services/webApi/app.js:199:15) at app.use.OpenApiValidator.middleware.apiSpec (/joystream/storage-node/lib/services/webApi/app.js:59:13) at Layer.handle [as handle_request] (/joystream/node_modules/express/lib/router/layer.js:95:5) at trim_prefix (/joystream/node_modules/express/lib/router/index.js:317:13) at /joystream/node_modules/express/lib/router/index.js:284:7 at Function.process_params (/joystream/node_modules/express/lib/router/index.js:335:12) at next (/joystream/node_modules/express/lib/router/index.js:275:10) at /joystream/storage-node/lib/services/webApi/app.js:54:9 at Layer.handle [as handle_request] (/joystream/node_modules/express/lib/router/layer.js:95:5) at trim_prefix (/joystream/node_modules/express/lib/router/index.js:317:13)

[dobertRowneySr]

I am not fully convinced that these error refers to the above objectId:

Background on upload file flow

1️⃣ Middleware chain

Currently on uploads we rely on the following middlewares chain:

1. verifyUploadFileParams middleware
2. OpenApiValidator.middleware with multer for uploads, into tempDirectory If 1 throws then 2 is not called.

2️⃣ Route handler

Once file upload has succesfully executed, the route handleruploadFile is caled and basically moves the file to the pendingDirectory after verifying the hash file

3️⃣ Accepting object service

Once every 6s (i.e. every target block production rate) the AcceptPendingObject service is called which processes objects in the pendingDirectory and then issues the accepting transaction for each of these. Only after the tx is successfully issued the objects are either moved into the uploadDir (from where they are actually served).

The errors describes appears to be Middleware errors (except for object 1713397) for which Express returns an error and the following steps 2️⃣ and 3️⃣ are not pursued.
Objects 1713397 encounters a timeout error in the acceptPendingDataObjects transaction.

[dobertRowneySr]

I am investigating on the first object and:

{
    "data": {
        "storageDataObjectById": {
            "isAccepted": true,
            "storageBag": {
                "id": "dynamic:channel:42488",
                "storageBuckets": [
                    {
                        "id": "10-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage2.sieemmastorage.com/storage/"
                            },
                            "id": "10"
                        }
                    },
                    {
                        "id": "1-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage.mrbovo.xyz/storage/"
                            },
                            "id": "1"
                        }
                    },
                    {
                        "id": "2-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://storage.joystream.8k.pm/storage/"
                            },
                            "id": "2"
                        }
                    },
                    {
                        "id": "3-dynamic:channel:42488",
                        "storageBucket": {
                            "operatorMetadata": {
                                "nodeEndpoint": "https://joystream.name/storage/"
                            },
                            "id": "3"
                        }
                    }
                ]
            },
            "acceptingBucketId": "12"
        }
    }
}

[dobertRowneySr]

I think this is worth a task on his own and not be included in the s3 epic as it might be require more time and investigation than needed