feat: Client copy - Issue 670

[b-eisen]

This code implements the feature request in #670. The code implements a number of new flags specific to the CP command to allow for copying from an external S3 compatible API to an S3 bucket in region. The implementation takes the existing download and upload methods, calling them in one function and saving, then deleting a file which is temporary on the local filesystem.

                &cli.BoolFlag{
			Name:    "client-copy",
			Aliases: []string{"cc"},
			Usage:   "copies from S3 bucket to S3 bucket through the local client (Advanced Use Case)",
		},
		&cli.StringFlag{
			Name:  "source-region-profile",
			Usage: "use the specified profile from the credentials file for the source region",
		},
		&cli.StringFlag{
			Name:  "destination-region-profile",
			Usage: "use the specified profile from the credentials file for the destination region",
		},
		&cli.StringFlag{
			Name:    "source-region-endpoint-url",
			Usage:   "override default S3 host for custom services for the source region",
			EnvVars: []string{"S3_ENDPOINT_URL"},
		},
		&cli.BoolFlag{
			Name:  "source-region-no-verify-ssl",
			Usage: "disable SSL certificate verification for the source region endpoint",
		},
		&cli.StringFlag{
			Name:    "destination-region-endpoint-url",
			Usage:   "override default S3 host for custom services for the destination region",
			EnvVars: []string{"S3_ENDPOINT_URL"},
		},
		&cli.BoolFlag{
			Name:  "destination-region-no-verify-ssl",
			Usage: "disable SSL certificate verification for the destination region endpoint",
		},

New E2E tests have been and all tests pass (except for TestRunSpecialCharactersInPrefix in run_test.go) which failed before new code was written. I have also performed integration testing on MacOS, Linux and Windows with both minio and S3 compatible API on Snowball Edge devices.

Make all fails on my local setup due to an issue with staticcheck. VSCode plugin shows no linting errors. It failed with

make: staticcheck: No such file or directory
make: *** [staticcheck] Error 1

[b-eisen]

@igungor @denizsurmeli - Good day. Following up to see if you have had a chance to look or have any questions on this PR. Thank you.

[b-eisen]

@igungor @denizsurmeli - Good day again. Following up to see if you have had a chance to look at this PR or have any questions. Thank you.

[denizsurmeli]

Hi, thanks for your work. This is a issue often mentioned for s5cmd. I would like to share my two cents about it, but obviously we can and will discuss with the team members about this feature. s5cmd is basically a file system tool. It's designed to operate on one filesystem. The feature you proposing is somehow using the host tool as a bridge, to sync two 'file systems'. There are separate tools that just designed for that purpose, such as rclone. This feature feels like out of the purpose of s5cmd, but again, this is a personal comment. Again, thanks for your contribution, we appreciate your work.

[b-eisen]

Thank you for your response. I understand that point of view and that it may not align with the purpose of s5cmd. I will note, however, that in my course of work I have been made away of users who want to continue to take advantage of the high performant nature of s5cmd while working with multiple s3 compatible endpoints. They have made workarounds that essentially wrap s5cmd in other tools (python scripts, etc.). This would allow users to have additional capability without needing the complexity of additional scripts, tools, etc.

In my firsthand experiences those users have looked at tools like rclone, but prefer the performance and familiarity of s5cmd.

Thank you for your consideration.

[b-eisen]

Good day. Following up to see if the team has discussed this issue yet?

[k0ste]

Actually it will be nice to see ability of s5cmd to work with two s3 endpoints. For example, when you need "add another region" (cp region-1 to new region-2) or move images from one region to another