gaffer01

Prototype and test transaction log based approach to the state store

1

### Background See related issues: - https://github.com/gchq/sleeper/issues/1355 - https://github.com/gchq/sleeper/issues/1700 ### Description See [design document](https://github.com/gchq/sleeper/blob/develop/docs/designs/transaction-log-state-store.md). We'd like to know whether the transaction log approach to the state store can improve on...

REST API

1

### Description Sleeper need a REST API to allow admin interactions, queries and data ingest requests to be sent via REST.

### Description We use SQS as a queue in many parts of the system, e.g. for compaction jobs, for ingest requests, etc. SQS works well in many ways, for examples...

Standardise on using AWS SDK version 2 everywhere

2

### Background The codebase currently uses a mixture of v1 and v2 of the AWS SDK. ### Description We would like to use v2 of the AWS SDK everywhere and...

### Background See this page for discussion of what an S3A directory marker is: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/directory_markers.md We have not explicitly set any properties relating to this. We have always implicitly used...

### Background We should investigate improvements to the query performance. It is normal in LSM stores to store a Bloom filter with each file. When querying for a key, the...

Error with CDK deployment

1

### Description Deploying Sleeper using CDK version 2.103.* normally works but displays an error message "Error occurred while monitoring stack: Error [ValidationError]: 2 validation errors detected: Value '' at 'stackName'...

### Background The property `sleeper.subnets` is not editable. ### Description A test was performed: The code in `CommonProperty` was changed so that the property `sleeper.subnets` is marked as editable and...

### Background We have the instance property `sleeper.log.retention.days` which controls the retention period for Cloudwatch logs. In some situations there may be account-level controls on the retention of logs, e.g....

We want to allow richer data to be stored in Sleeper, specifically data with fields which consist of structs, arbitrarily nested structs, etc. Both Parquet and Arrow support this, and...