Jason Carneiro

Elasticsearch ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: elasticsearch namespace: conductor labels: app: elasticsearch spec: selector: matchLabels: app: elasticsearch replicas: 1 template: metadata: labels: app: elasticsearch spec: containers: - name:...

Follows the Amazon State Language specification https://states-language.net/spec.html#errors

Another test ``` chi-signoz-tools-cluster-clickhouse-cluster-0-0-0:~$ ./clickhouse-backup -c config.yml tables signoz_metrics.samples_v2 11.97GiB default full signoz_metrics.samples_v4 10.01GiB default,s3 full chi-signoz-tools-cluster-clickhouse-cluster-0-0-0:~$ ./clickhouse-backup -c config.yml create_remote partial 2024-10-10 21:55:48.653 INF pkg/backup/create.go:170 > done createBackupRBAC size=0B...

> Did you setup AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY inside clickhouse-backup container? I running clickhouse-backup bin inside clickhouse-server container. The used service account works for normal clickhouse-server workloads (s3 disk as cold...

I changed `/var/lib/clickhouse/preprocessed_configs/config.xml` file adding aws credentials and the warning is not shown, but access denied error remains ``` 2024-10-11 14:40:56.684 INF pkg/backup/create.go:170 > done createBackupRBAC size=0B 2024-10-11 14:40:56.735 WRN...

See the code block [here](https://github.com/Altinity/clickhouse-backup/blob/master/pkg/storage/s3.go#L485) Is the `srcBucket` variable empty? Compare the output log ``` error="one of createBackupLocal go-routine return error: one of uploadObjectDiskParts go-routine return error: b.dst.CopyObject in /var/lib/clickhouse/disks/s3/backup/2024-10-11-remote2/shadow/signoz_logs/logs/s3...

The s3 logs ``` 2024-10-11 15:16:27.034 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request GET /?versioning= HTTP/1.1 Host: data2.s3.xxxxxxxxxxxxx-cold-storage-tools.amazonaws.com User-Agent: m/F aws-sdk-go-v2/1.30.5 os/linux lang/go#1.22.7 md/GOOS#linux md/GOARCH#arm64 api/s3#1.61.2 Accept-Encoding: identity Amz-Sdk-Invocation-Id: 338f4e3a-00cd-4f38-a096-ea36114e0b97 Amz-Sdk-Request: attempt=1;...

The s3 endpoint string format was wrong. I changed `https://xxxxxx-storage-test-tools.s3.amazonaws.com/data/` to `https://xxxxxx-storage-test-tools.s3.us-east-1.amazonaws.com/data/` and work

No. I installed clickhouse-backup bin manually in the container

I think to fork the chart and customize to provide side cars containers to clickhouse-server. About embbeded backup suggestions, I tried but the backup fails for clustered workload and I...