Oleg Avdeev

I think we need to make it backwards compatible so existing configs with `METAFLOW_DATATOOLS_S3ROOT` continue to work, and then merge

Left some comments, sorry it took so long to get to it. But we definitely need a fix, even our internal tests fail due to this once in a while....

Nice catch! The only thing that gives me a bit of a pause here is that it is not uncommon to have `:` in env variable values (eg `$PATH`). On...

After testing this a bit more, this doesn't seem to work with `--batch`. In that case gets converted back to `--with` CLI arg [here](https://github.com/Netflix/metaflow/blob/master/metaflow/decorators.py#L112), and it turns it into `--with...

Something like this https://github.com/Netflix/metaflow/commit/d654bc8727b18f88c27e90bb95288a0f52ff8c3c should work but may still need a bit more testing

@savingoyal had a suggestion that we could solve it in a slightly more elegant way by making `@environment` a "varargs" style decorator. So instead of `@environment(vars={"FOO": "BAR"})` you could support...

See also https://github.com/Netflix/metaflow/pull/768

Note that there is already mechanism in botocore to refresh credentials from env vars automatically. Really the problem is specific to Cloud9 which chose a weird way to provision temporary...

Yep, I was thinking `@docker` at first, but then remembered that it would be technically inaccurate. In the list above only AWS Batch uses Docker per se, others use containerd...

As I think more about it, maybe it is a good idea to borrow the terminology from [k8s container spec](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Container.md)? In theory, we may decide to support overriding `working_dir` too,...