Cannot specify namespace to use for label selectors with `k8s_custom_deploy`

[milas]

Expected Behavior

• Possible to rely on label selectors with k8s_custom_deploy that does not return any YAML from apply_cmd

Current Behavior

• We only watch for events in namespaces that we've seen a deployed object in, which we determine from result YAML, so if we never get any YAML, we won't watch any namespaces and label selectors will silently fail
  • If you also apply resource(s) that DO use result YAML, assuming there's a namespace overlap, things will work implicitly

Steps to Reproduce

1. Create a Tiltfile with a k8s_custom_deploy that doesn't return YAML and relies on label selectors:

    k8s_custom_deploy(
        'nginx',
        apply_cmd='kubectl apply -f app.yaml 1>&2',
        delete_cmd='kubectl delete -f app.yaml',
        deps=['app.yaml']
    )
   
    k8s_resource('nginx', extra_pod_selectors={'app': 'web'}, discovery_strategy='selectors-only')
   

   app.yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web
    spec:
      selector:
        matchLabels:
          app: web
      replicas: 1
      template:
        metadata:
          labels:
            app: web
        spec:
          containers:
            - name: web
              image: nginx
              resources:
                limits:
                  cpu: 100m
                  memory: 128Mi
              ports:
                - containerPort: 80
              readinessProbe:
                httpGet:
                  port: 80
                failureThreshold: 1
                periodSeconds: 10
   

2. Run tilt up

3. Run kubectl get pod -l app=web to see that pod exists and is healthy

4. Observe that Tilt UI shows it as perpetually pending / has no logs / is unaware of Pod's existence

Context

I think the most logical thing here is to add an optional namespace arg to k8s_resource to explicitly use in the KubernetesDiscoveryTemplateSpec + KubernetesDiscoverySpec in addition to(?) any implicitly discovered namespaces. It should probably be an error to set this without also populating extra_pod_selectors because otherwise it's meaningless.

Alternatively, we could have a "magic" key in extra_pod_selectors that's something like $namespace (not a valid K8s label identifier, so no risk of overlap) to use for this purpose without changing the API.

We also currently eagerly watch any namespaces based on the spec YAML; in the case of a spec apply cmd, we could eagerly watch the "default" namespace in that case, but that only works if that's the namespace the apply cmd actually uses, which isn't a given.

[emanuele-leopardi]

I've got a workaround if you want , you could specify the function

def deploy_yaml(service,namespace=default):
  k8s_custom_deploy(
    service,
    apply_cmd=str('kubectl apply -f {service}.yaml -n {namespace} -o yaml'.format(service=service,namespace=namespace)),
    delete_cmd=str('kubectl delete -f {service}.yaml -n {namespace}'.format(service=service,namespace=namespace)),
    deps=[service],
    )

and call it as

deploy_yaml("my-service","my-namespace")

notice the very important -o yaml in the apply command The UI shows this

Deploying
     Running cmd: kubectl apply -f dep/my-service.yaml -n my-namespace -o yaml
     Objects applied to cluster:
       → my-service:deploy