s3proxy icon indicating copy to clipboard operation
s3proxy copied to clipboard

Using Apache Camel or S3Client is writing additional content compared to CLI to storage

Open ZeroWiggliness opened this issue 3 months ago • 0 comments

I have a weird issue where we see that writing files s3Proxy is also adding chunk-signatures and checksums to the file content when using either Apache Camel or S3Client (which the Camel connector is based on). Using the cli is totally fine. The same code writes files to S3 and localstack as I would expect.

We would prefer s3 proxy as we typically use Azure but we see the same with the filesystem configuration.

File content: "fileContent-none"

CLI

Command: docker run --rm -ti -e AWS_ACCESS_KEY_ID=dummy -e AWS_SECRET_ACCESS_KEY=dummy -e AWS_DEFAULT_REGION=eu-central-1 -it -v .:/aws amazon/aws-cli s3 cp /aws/test.txt s3://etl-export-container/ --endpoint-url http://host.docker.internal:4560 Content in file: "fileContent-none"

Camel/S3Client

Code: PutObjectRequest request2 = PutObjectRequest.builder() .bucket(BLOB_CONTAINER_NAME) .key(filename + "s3client") .contentType("text/plain; charset=UTF-8") .build();

s3Client.putObject(request2, RequestBody.fromString(fileContent));

Content in file: "10;chunk-signature=460c0b1b7dfc968c3c1bfbdff125747c5afb0afa0ea3129f63b9a0dcf1205bc2 fileContent-none 0;chunk-signature=65183ac24cbe0eed3b56470c462bae1f70e04b464da0caed2a4df18d44e26bdb x-amz-checksum-crc32:VdfJwQ== x-amz-trailer-signature:80866cca3e043de1df9ee970273eec29cb69be8da7cd9f6c16c3fef2cb77cd25 "

We deploy via docker with locally: Image: andrewgaul/s3proxy:sha-6355f7826b8e25b55067b9713d852c4d8e41e0ca

  S3PROXY_ENDPOINT: "http://0.0.0.0:4560"
  S3PROXY_AUTHORIZATION: "none"
  S3PROXY_IGNORE_UNKNOWN_HEADERS: "true"
  LOG_LEVEL: "info"
  JCLOUDS_PROVIDER: "azureblob-sdk"
  #JCLOUDS_PROVIDER: "filesystem-nio2"
  JCLOUDS_IDENTITY: "devstoreaccount1"
  JCLOUDS_CREDENTIAL: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
  JCLOUDS_ENDPOINT: "http://azurite:10000/devstoreaccount1"
  JCLOUDS_AZUREBLOB_AUTH: "azureKey"
  JCLOUDS_FILESYSTEM_BASEDIR: "/data"

That I can see there should be no reason for the extra data.

Any help is appreciated.

ZeroWiggliness avatar Nov 05 '25 16:11 ZeroWiggliness