dora-team
Support installation on Kubernetes
Many users have asked for the ability to install Four Keys onto Kubernetes. This requires (at least):
- Use some replacement for Pub/Sub. Maybe an open source pub/sub tool running in Kubernetes?
- Run the event-handler and data-parsers in Kubernetes. These are already container based images so they should fit in nicely.
- Replace BigQuery with a vanilla SQL database running Kubernetes
- Update the API calls in the services to use the open source equivalents of Pub/Sub and BigQuery.
It would be great to see that project run on other stacks as well.
For K8s what were you thinking:
- Rabbitmq, kafka
- event handler and parser will just be deployment
- big query could be using Druid
Yes, that's in line with what I was thinking, @emirot -- would you be interested in collaborating on this?
I would like to use this project at my current job, (if so I might have some time) however the decision won't be mine.
Apache Pulsar could be a good fit for a pub/sub replacement? There's a helm chart for easy deployment here: https://pulsar.apache.org/docs/en/helm-deploy/
There's an open PR for Pulsar integration with Druid that just needs a bit of work. The built in Trino (formerly Presto) SQL interface supported by Pulsar isn't supported by Grafana.
Being that this project already uses Knative (CloudRun), Knative Eventing would be a solid choice.
