[BUG] specifying image in PodTemplate does not set the image used for a NotebookTask

[blair-anson]

Describe the bug

I have been using a PodTemplate to set the default docker image for a normal @task(), as per the link below. However the NotebookTask ignores the image setting in the PodTemplate, although it does apply other settings such as VolumeMount

https://docs.flyte.org/en/latest/deployment/configuration/general.html#using-default-k8s-podtemplates

Example of a PodTemplate that is generated by clusterResourceTemplates in the values.yaml

$ kubectl get podtemplate flyte-template -n test8-development -o yaml
apiVersion: v1
kind: PodTemplate
...
template:
...
  spec:
    containers:
    - image: xxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-own-image:my-tag
      imagePullPolicy: IfNotPresent
      name: default
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
...

Example of the Pod that runs

$ kubectl get pod f05d6a5f90403454f88b-n0-0 -n test8-development -o yaml
apiVersion: v1
kind: Pod
...
    image: cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1
    imagePullPolicy: IfNotPresent
    name: f05d6a5f90403454f88b-n0-0
...
  containerStatuses:
  - containerID: docker://adda36ed3823592a117e5b99f284accddee987e5d966e6b48b1c46eab7e98fa7
    image: cr.flyte.org/flyteorg/flytekit:py3.8-1.4.1
    imageID: docker-pullable://cr.flyte.org/flyteorg/flytekit@sha256:dcc3c40249bb983eb96dc0024f1506f4e261a64f1c3a1f447f94750720cb8ae3
    lastState: {}
    name: f05d6a5f90403454f88b-n0-0
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
...

Expected behavior

The Pod processing the NotebookTask should use the docker image defined in the PodTemplate

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

"Hello 👋, this issue has been inactive for over 90 days. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏"

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 90 days and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏