Tracking issues of RFC-0423: Command Line Interface

[Xuanwo]

• https://github.com/datafuselabs/opendal/pull/423

oli {cp,rm,ls,...}
ocp s3://abc azblob://def
orm s3://abc
ols s3://abc

Todo List

We will implement a quick demo that only supports cp from local to remote.

• [x] Add a basic project layout
• [x] https://github.com/datafuselabs/opendal/issues/675

Future

• [ ] Full cp support
• [x] mv
• [x] stat
• [x] cat
• [x] tee
• [x] rm
• [x] ls
• [ ] Progress bar

[Xuanwo]

We can use opendal to replace tools like aws-cli, s3cmd, azcopy, and so on, making it easy for tests.

[teckick]

/assignme

[Xuanwo]

This issue is too general. @eastfisher please feel free to create sub-issues and link here.

[RinChanNOWWW]

How about adding a ALIAS subcommand to associate with the storage service, so we don't need to input the whole URL every time.

Like mc(MinIO Client): https://min.io/docs/minio/linux/reference/minio-mc.html#create-an-alias-for-the-s3-compatible-service

[RinChanNOWWW]

Oh, it seems that profile can handle this.

[Xuanwo]

Hi, @Zheaoli. Maybe you wanna take a look at this issue.

After hard working from @eastfisher, oli is able to do simple copies between different services.

And in next stage, we will to implement:

• ols [-r] s3://my_bucket
• ocp -r s3://my_bucket gcs://my_bucket (needs ols support)
• orm [-r] s3://bucket/path/to/file (needs ols support)

After those work, we can try to replace the usage of s3cmd in some real user cases and to explore what's the most important next.

[knight42]

@Xuanwo Hi, oli seems to be an awesome tool 🚀 I would like to get involved in the development!

Before I can start, I would like to introduce a config file to oli. IMHO a single config file is easier to manage than a handful of env vars.

The config file consists of multiple profiles, each profile represents a service users want to access. Users are able to specify the endpoint, credentials and other necessary parameters in each profile. The config file might look like this(the format doesn't matters, I just prefer toml now):

# ~/.oli/config.toml

[[profiles]]
name = "mys3" # required
service = "s3" # required
endpoint = "" # optional
access-key = "<redacted>"
secret-key = "<redacted>"

[[profiles]]
name = "myaz" # required
service = "azblob" # required
endpoint = "" # optional
access-key = "<redacted>"
secret-key = "<redacted>"

When users want to interact with multiple services, like copying objects across services, they need to run

oli cp mys3://bucket/1.txt myaz://bucket2/1.txt

Please note that the profile name is used as scheme to distinguish different services here.

I think the advantages of the config file over env vars including but not limited to:

1. it is easier to manage a config file than exporting a lot of env vars in a shell rc file
2. the config file can be strongly typed(like toml), it is easier to validate the config
3. it is more straightforward to see which service has been configured with a config file

@Xuanwo WDYT?

[Xuanwo]

LGTM!

[knight42]

LGTM!

Wow I didn't expect to get an instant reply 😂 I would like to work on the config schema definition and loading logic first.

[Zheaoli]

@Xuanwo Hi, oli seems to be an awesome tool rocket I would like to get involved in the development!

Before I can start, I would like to introduce a config file to oli. IMHO a single config file is easier to manage than a handful of env vars.

The config file consists of multiple profiles, each profile represents a service users want to access. Users are able to specify the endpoint, credentials and other necessary parameters in each profile. The config file might look like this(the format doesn't matters, I just prefer toml now):

# ~/.oli/config.toml

[[profiles]]
name = "mys3" # required
service = "s3" # required
endpoint = "" # optional
access-key = "<redacted>"
secret-key = "<redacted>"

[[profiles]]
name = "myaz" # required
service = "azblob" # required
endpoint = "" # optional
access-key = "<redacted>"
secret-key = "<redacted>"

When users want to interact with multiple services, like copying objects across services, they need to run

oli cp mys3://bucket/1.txt myaz://bucket2/1.txt

Please note that the profile name is used as scheme to distinguish different services here.

I think the advantages of the config file over env vars including but not limited to:

1. it is easier to manage a config file than exporting a lot of env vars in a shell rc file
2. the config file can be strongly typed(like toml), it is easier to validate the config
3. it is more straightforward to see which service has been configured with a config file

@Xuanwo WDYT?

I think we can hide the bucket from the path and move it into the config profile?

[Xuanwo]

I think we can hide the bucket from the path and move it into the config profile?

Yes, this way we no longer need to handle the bucket specially.

[knight42]

I think we can hide the bucket from the path and move it into the config profile?

Yes, this way we no longer need to handle the bucket specially.

I am hesitant to add the bucket to the config file, as users might want to access different buckets in the same account. As a user, I don't want to create different profiles for different buckets in the same account.

[knight42]

@Xuanwo Do you think it is necessary to open separate issue to track the implementation of each command?

[Xuanwo]

@Xuanwo Do you think it is necessary to open separate issue to track the implementation of each command?

I think it's a good idea!