[FLINK-32315][k8s] Support uploading "local://" artifacts in Kubernetes Application Mode

[ferenc-csaky]

What is the purpose of the change

Makes it possible to upload local:// artifacts in K8s Application Mode to remote DFS, so combined with Artifact Fetching, it is not necessary to bundle all Flink job deps to the Docker image.

Upload handles the job JAR and any additional local artifact that is present in the user.artifacts.artifact-list config. The user.artifacts.artifact-list can contain both local and remote artifact mixed, former will be uploaded, latter will remain unchanged and everything will be fetched on the JM pod.

Example

# Without additional artifacts.
$ ./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    -Dkubernetes.artifacts.local-upload-enabled=true \
    -Dkubernetes.artifacts.local-upload-target=s3://my-bucket/ \
    local:///tmp/my-flink-job.jar

# With additional artifacts.
$ ./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster \
    -Dkubernetes.container.image=custom-image-name \
    -Dkubernetes.artifacts.local-upload-enabled=true \
    -Dkubernetes.artifacts.local-upload-target=s3://my-bucket/ \
    -Duser.artifacts.artifact-list=local:///tmp/my-flink-udf1.jar\;s3://my-bucket/my-flink-udf2.jar \
    local:///tmp/my-flink-job.jar

Brief change log

• Added new KubernetesConfigOptions for uploading.
• Introduced KubernetesArtifactUploader and a default impl, which handles the uploading.
• Wired in the uploading to KubernetesClusterDescriptor.

Verifying this change

• Added unit tests in DefaultKubernetesArtifactUploaderTest.
• Verified the change manually with Minikube and Minio.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs

[flinkbot]

CI report:

• 5047c3f7e09387eafac375cae0f0f84337b20328 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[ferenc-csaky]

@flinkbot run azure