Prefect K8S agent error when deploying a flow run

[aidenprice]

Description

I get an occasional error on my Kubernetes agent when deploying a flow run. I have noticed that the flow doesn't start or will sometimes start much later.

This error is roughly correlated with the time I started trying to deploy to our existing Dask cluster, and to the time when we upgraded our Kubernetes version.

[2021-01-17 00:00:11,063] ERROR - AGENTNAME | Error while managing existing k8s jobs
Traceback (most recent call last):
  File "/usr/local/.venv/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 362, in heartbeat
    self.manage_jobs()
  File "/usr/local/.venv/lib/python3.8/site-packages/prefect/agent/kubernetes/agent.py", line 219, in manage_jobs
    event.last_timestamp
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'

The current workaround is to click the run button again.

Expected Behavior

The flow should be deployed and begin processing at the earliest opportunity.

Reproduction

I don't have a great deal of insight into what the underlying cause is.

Environment

Prefect server stack deployed to Kubernetes with a modified version of the official Helm chart. I have a custom docker image and an ingress.

Our cluster is running in Azure Kubernetes Service, Kubernetes version 1.17.13