Ville Tuulos

A way to clean up old runs / artifacts from datastore

6

There isn't an easy way to clean up old entries from the datastore. S3 lifecycle policies work for metadata but not for content-addressed artifacts, which may be shared by multiple...

Better error message for artifact serialization errors

2

When an artifact fails to serialize, we should show a clear error message highlighting the specific artifact that caused trouble. Context: https://gitter.im/metaflow_org/community?at=5e29e840348292727916743a

Make it possible to disable certain options

1

A system administrator may want to disable an option, e.g. `--max-workers` to enforce a system-wide setting. We could provide a way to disable selected options, or override any settings globally....

Expiring AWS credentials

4

In some environments like AWS Cloud9 or `aws-vault`, there's a background process that updates AWS credentials at ~/.aws/credentials (or env vars) periodically. Currently, Metaflow/`boto` don't refresh these credentials, leading to...

create a minimal Docker container for TrailDB

1

a'la https://blog.codeship.com/building-minimal-docker-containers-for-go-applications/

To guard against bitrot

We should add an `info` command in tdb cli which prints out basic stats about TrailDB quickly

TrailDB could handle high-cardinality fields more efficiently. We have faced two examples of high-cardinality fields recently: - IDs of format `granular_timestamp + random ID` (e.g. `144500000009837478`) - Continuous-valued fields with...

## Problem The current API requires that you set the cursor explicitly for each trail using `tdb_get_trail`. When combined with event filters, it is possible that some trails yield no...

Every release that changes the on-disk format of TrailDB should add a set of TrailDBs to a directory that is stored as binaries with `git lfs`. We should have a...