tusd icon indicating copy to clipboard operation
tusd copied to clipboard
verified publisher icon tus

randomly SignatureDoesNotMatch error on upload large file (>50MB)

Open mosi-kha opened this issue 3 years ago • 7 comments

Describe the bug on upload file (>50MB) raise the error SignatureDoesNotMatch on middle of progess also do not mention s3store on detail. these are the last 10 tusd's log:

[tusd] 2022/07/25 13:06:02 event="ChunkWriteComplete" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" bytesWritten="500000" 

[tusd] 2022/07/25 13:06:02 event="ResponseOutgoing" status="204" method="PATCH" path="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" requestId="" 

[tusd] 2022/07/25 13:06:02 event="RequestIncoming" method="PATCH" path="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" requestId="" 

[tusd] 2022/07/25 13:06:02 event="ChunkWriteStart" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" maxSize="500000" offset="48500000" 

[tusd] 2022/07/25 13:06:03.960933 event="HookInvocationStart" type="post-receive" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" 

[tusd] 2022/07/25 13:06:03.969287 event="HookInvocationFinish" type="post-receive" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" 

[tusd] 2022/07/25 13:06:04.961186 event="HookInvocationStart" type="post-receive" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" 

[tusd] 2022/07/25 13:06:04.968753 event="HookInvocationFinish" type="post-receive" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" 

[tusd] 2022/07/25 13:06:35 event="ChunkWriteComplete" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" bytesWritten="0" 

[tusd] 2022/07/25 13:06:35 event="ResponseOutgoing" status="500" method="PATCH" path="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" error="SignatureDoesNotMatch: 
        status code: 403, request id: tx00000e18bb708b10ffb71-0062de5613-158baee2-default, host id: " requestId=""

To Reproduce Steps to reproduce the behavior:

  1. Export AWS access keys
  2. Run server: tusd --hooks-http http://127.0.0.1:8000/upload/tusd -s3-bucket=tusd -s3-endpoint=https://s3.ir-thr-at1.arvanstorage.com
  3. Snippet for upload file with tuspy:
from tusclient import client
from tusclient.storage import filestorage

storage = filestorage.FileStorage("storage_file")

my_client = client.TusClient(url="http://0.0.0.0:1080/files/")
my_uploader = my_client.uploader(
    "/path/to/file", # bigger than 50 MB
    chunk_size=500000,
    store_url=True,
    url_storage=storage,
)
my_uploader.upload()
  1. See error on middle of upload prgress: [tusd] 2022/07/25 13:06:35 event="ResponseOutgoing" status="500" method="PATCH" path="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" error="SignatureDoesNotMatch: status code: 403, request id: tx00000e18bb708b10ffb71-0062de5613-158baee2-default, host id: " requestId=""

Expected behavior A clear and concise description of what you expected to happen.

Setup details Please provide following details, if applicable to your situation:

  • Operating System: kubuntu 22.04
  • Used tusd version: 1.9.0
  • Used tusd data storage: AWS S3
  • Used tusd configuration: --hooks-http http://127.0.0.1:8000/upload/tusd -s3-bucket=tusd -s3-endpoint=https://s3.ir-thr-at1.arvanstorage.com
  • Used tus client library: tuspy

mosi-kha avatar Jul 25 '22 10:07 mosi-kha

-s3-endpoint=https://s3.ir-thr-at1.arvanstorage.com

It seems you are not actually using AWS S3 but an S3-compatible storage. Is that correct? It is also possible that this is a bug in their S3 implementation as we have never seen this error for our usage of tusd with AWS S3.

The logs also look odd to me. It's quite possible that there is already something problematic going on before this error.

[tusd] 2022/07/25 13:06:35 event="ChunkWriteComplete" id="e4cf9cf969f3d6563d0596747bb6f2a7+2~WSdh726Nr02zF1QhEgGpzWf6B9Lq1QR" bytesWritten="0"

This says that no bytes were written to S3?

Does the upload work if you configure tuspy to use the https://tusd.tusdemo.net/files/ endpoint? To rule out a problem with your tuspy configuration?

Acconut avatar Jul 25 '22 10:07 Acconut

It seems you are not actually using AWS S3 but an S3-compatible storage. Is that correct? It is also possible that this is a bug in their S3 implementation as we have never seen this error for our usage of tusd with AWS S3.

Yes, we use ArvanCloud object storage that s3 compatible, we use it in our production a lot. we want use tusd as new upload mechanism so in my tests this error raised.

Does the upload work if you configure tuspy to use the https://tusd.tusdemo.net/files/ endpoint?

I didn't test with this endpoint. I'll try with it right now.

mosi-kha avatar Jul 25 '22 11:07 mosi-kha

uploaded successfully on https://tusd.tusdemo.net/files/.

In my error case, If I upload the same file again, it will continue to upload the errored place and finish it.

mosi-kha avatar Jul 25 '22 11:07 mosi-kha

uploaded successfully on https://tusd.tusdemo.net/files/.

Ok, this indicates that there is no issue with your client configuration. However, this indicates that there is probably a difference in behaviour between AWS SE and ArvanCloud.

In my error case, If I upload the same file again, it will continue to upload the errored place and finish it.

Ok, so this means that you are able to complete the upload successfully if you resume?

Can you provide more logs if an upload fails? In the best case all logs since the beginning of the upload.

Also, please try increasing the chunk size setting in tuspy. Maybe try 10 MB.

Acconut avatar Jul 28 '22 10:07 Acconut

Ok, so this means that you are able to complete the upload successfully if you resume?

yes, I can.

Can you provide more logs if an upload fails? In the best case all logs since the beginning of the upload.

(moved to https://gist.github.com/Acconut/9d5ae1a952e9917671a84a50ead172ef)

Also, please try increasing the chunk size setting in tuspy. Maybe try 10 MB.

it's okey.

mosi-kha avatar Jul 31 '22 12:07 mosi-kha

it's okey.

Does that help with the uploading issues of SignatureDoesNotMatch?

I have also found a report about this error in conjunction with network issues: https://github.com/aws/aws-sdk-go/issues/3237 Could it be that you have connectivity issues and these manifest as signature errors?

Acconut avatar Aug 01 '22 10:08 Acconut

Does that help with the uploading issues of SignatureDoesNotMatch?

yes, it's not happened with bigger chunk sizes in my couple of tests.

aws/aws-sdk-go#3237 Could it be that you have connectivity issues and these manifest as signature errors?

Hmm, that makes sense, we have network connection issues a lot, do you have any suggestion or solution for it?

mosi-kha avatar Aug 02 '22 06:08 mosi-kha