Ability to add or manage labels, annotations, node selection, and tolerations on KFP pods

[RyanRennCN]

Checks

• [X] I have searched the existing issues.
• [X] This request is NOT specific to the CLI. (Please open a CLI issue)

Motivation

I recently had a need to restrict our cluster auto-scaler from scaling down nodes that have in progress KFP pipeline pods running on them. The method for restricting scale down was via a label that has to be added to all of the KFP pipeline pods. While the kfp-kubernetes package allows you to label tasks, it does not seem to have a way to label the entire pipeline in KFP V2.

In order to avoid duplicating code for every task and to avoid having to update all pipelines, it would be great if there was a way to natively add labels, tolerations, node selectors, and annotations to all KFP pipeline pods via the deployKF config.

Implementation

In the app of apps yaml config, it would be great if there was a section where you could add these items and have it applied to all kfp pipeline pods.

For my case, I have a workaround in place where I have an "overlay" Helm chart I apply to the cluster and the labels are added via a Kyverno cluster policy.

Are you willing & able to help?

• [X] I am able to submit a PR!
• [X] I can help test the feature!

[thesuperzapper]

@RyanRennCN this is a great idea, and we already have a Kyverno ClusterPolicy which mutates all the KFP Pods, so we can simply provide some additional deployKF values which allow setting arbitrary Annotations/Labels:

• https://github.com/deployKF/deployKF/blob/v0.1.5/generator/templates/manifests/kubeflow-tools/pipelines/resources/profile-resources.yaml#L198-L261

However, I am not sure it's a good idea for all Workflow Pods to have the same node-selectors/tolerations as these should be managed by Kubeflow Pipelines itself, on a per-pipeline-step basis.

[RyanRennCN]

@RyanRennCN this is a great idea, and we already have a Kyverno ClusterPolicy which mutates all the KFP Pods, so we can simply provide some additional deployKF values which allow setting arbitrary Annotations/Labels:

• https://github.com/deployKF/deployKF/blob/v0.1.5/generator/templates/manifests/kubeflow-tools/pipelines/resources/profile-resources.yaml#L198-L261

However, I am not sure it's a good idea for all Workflow Pods to have the same node-selectors/tolerations as these should be managed by Kubeflow Pipelines itself, on a per-pipeline-step basis.

@thesuperzapper - thanks. Would the labels and annotations be something I could attempt to add myself with a PR?

As for the node selectors and tolerations, what is the correct way to handle that in KFP?

Thanks