xpk
xpk copied to clipboard
[DO NOT MERGE] [Only for the long running PW job] Pathways gcsfuse intergation
Fixes / Features
- This enables memory mounted gcsfuse buckets for the large scale cluster
- Changes the startup policy of the Pathways pods
- Removes podFailurePolicy from pathways containers
Testing / Documentation
Verified on a large scale run
The startupPolicy and podFailurePolicy related changes can be merged into main
The gcsfuse related changes are not ready to be merged into main and needs further work:
- We need to integrate with the xpk storage API to properly plumb gcsfuse related volumes/volume mounts/annotations into PW container definitions.
- The changes rely on PV and PVC resources to be already present on the cluster. The storage API integration would also ensure that those resources are created by XPK automatically