xpk icon indicating copy to clipboard operation
xpk copied to clipboard

[DO NOT MERGE] [Only for the long running PW job] Pathways gcsfuse intergation

Open shauryagup opened this issue 11 months ago • 0 comments

Fixes / Features

  • This enables memory mounted gcsfuse buckets for the large scale cluster
  • Changes the startup policy of the Pathways pods
  • Removes podFailurePolicy from pathways containers

Testing / Documentation

Verified on a large scale run

The startupPolicy and podFailurePolicy related changes can be merged into main

The gcsfuse related changes are not ready to be merged into main and needs further work:

  1. We need to integrate with the xpk storage API to properly plumb gcsfuse related volumes/volume mounts/annotations into PW container definitions.
  2. The changes rely on PV and PVC resources to be already present on the cluster. The storage API integration would also ensure that those resources are created by XPK automatically

shauryagup avatar Mar 02 '25 22:03 shauryagup