Custom Backend Client Multi Part Upload not finding uploaded parts

[lukas-klammt-mw]

Hey,

I got a question concerning the multi part upload for a custom backend client and hope someone here can help.

I want to assemble the user uploaded parts in the zenko cloudserver and send them to my backend as one file. Sadly I am getting an error creating the filteredPartsObj in the completeMPU method. The issue is, that the storedParts array in the mdInfo object, passed to my completeMPU method is empty. The jsonList object passed to my completeMPU contains the uploaded parts ETag and part number correctly.

I am able to assemble the parts in the completeMPU method and send them to my backend. But I am still getting an error because the storedParts could not be found and the zenko cloudserver can't complete correctly.

This is the the printed filteredPartsObj after calling myfilteredPartsObj = validateAndFilterMpuParts(storedParts.Contents, jsonList, mpuOverviewKey, splitter, log); in my completeMPU method (see below): {"partList":[],"error":{"code":400,"description":"One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.","InvalidPart":true}}

My clients uploadPart method:

uploadPart(request, streamingV4Params, stream, size, key, uploadId, partNumber, bucketName, log, callback){
      let hashedStream = stream;
      if (request) {
          const partStream = prepareStream(request, streamingV4Params,
              log, callback);
          hashedStream = new MD5Sum();
          partStream.pipe(hashedStream);
      }
      let file = fs.createWriteStream(key+partNumber);
      hashedStream.pipe(file);
      let globKey = key;
      file.on('finish', () => {
        const dataRetrievalInfo = {
              key: bucketName+'/'+key,
              dataStoreETag: md5File.sync(path.resolve(globKey+partNumber)),
              dataStoreType: 'mystore',
              dataStoreName: 'mystore'
        };
       return callback(null, dataRetrievalInfo);
      });
  }

My clients completeMPU method:

completeMPU(jsonList, mdInfo, key, uploadId, bucketName, log, callback) {
      let filteredPartsObj;    
      const { storedParts, mpuOverviewKey, splitter } = mdInfo;
      filteredPartsObj = validateAndFilterMpuParts(storedParts.Contents, jsonList,
           mpuOverviewKey, splitter, log);
       const files = fs.readdirSync('.').filter(a => a.startsWith(key));
       const parts = [];
       for(const file of files)
             parts.push(fs.readFileSync(file));

        fs.writeFileSync(key, Buffer.concat(parts));
        let rs = fs.readFileSync(key);

        fetch(`${this.endpoint}/object/${bucketName}/${key}`, { headers: {'Authorization': 'Bearer SOMETHINGHARDCODED'}, method: 'PUT', body: rs }).then((response) => {
              if(response.status === 200) {
                 const completeObjData = {
                    key: bucketName+'/'+key,
                   //add in filteredObj here,
                    eTag: this._getEtag(jsonList.Part) 
                  };
                return callback(null, completeObjData);
              }
              else {
                return callback(this._getError(response.status));
              }
        });
    }
  }

_getEtag(partList){
    let eTag='';
    let concat='';
    partList.forEach(partObj => {
      concat+=partObj.ETag[0];
    });
    eTag=crypto.createHash('md5').update(concat).digest("hex");
    eTag+='-'+partList.length;
    return eTag;
  }

Am I missing something? Are the storedParts retrieved from the ShadowBucket? If so, what exactly is stored in the ShadowBucket and do i need to write to it myself or is the ShadowBucket completely handled by the existing logic of the zenko cloudserver?

Hope someone can help. Thanks in advance! :)

Kind regards Lukas