Mykhailo Osypov

Retry every S3 request on 5xx server errors

2

The 5xx server errors for s3 have to be retried, however rusoto has no such feature. See discussion https://github.com/delta-io/delta-rs/pull/510#issuecomment-981887975 for more details.

# Description For S3 we have localstack to mock s3/dynamodb usages. But not for others. For GCS quick googling resulted in https://github.com/fsouza/fake-gcs-server. We might give it a try. It also...

Recover from S3 rename/commit pauses

2

# Description When S3 pauses during a commit to the extent that the lock is expired and acquired by other worker then the commit is returned as failed. But the...

# Description Currently we copy only single table from [golden dataset](https://github.com/delta-io/connectors/tree/master/golden-tables/src/test/resources/golden) and use it in tests. Eventually it'd make sense to use every table from golden dataset and either use...

Keep dead_letters table/client on demand

1

We don't want to waste resources with dead_letters table and s3/dynamodb clients. Thus, it makes more sense to create/close those resources on demand

See https://github.com/delta-io/delta-rs/pull/434#issuecomment-922351666

Consumer cannot reseek back to offset 0 on conflict resolution

1

``` 2021-08-19T16:38:45 kafka_delta_ingest [INFO] - WORKER-1: pre-process message 1:10 -- id:46 2021-08-19T16:38:46 kafka_delta_ingest::instrumentation [INFO] - WORKER-1: Delta write for version 7 has completed in 414 millis 2021-08-19T16:38:46 kafka_delta_ingest [INFO] -...

Debug is inreadable now, for example ``` DEBUG kafka_delta_ingest::instrumentation] StatsHandler received stat MessageTransformed with value 1 DEBUG kafka_delta_ingest::instrumentation] StatsHandler awaiting channel. ``` and more