Maintaining deployment of core services on multiple layers

[joshmue]

The deployment of some services should be possible on multiple "layers" of SCS infrastructure. E. g. deploying Prometheus on "top level" payload K8s clusters, but also on "low level" infrastructure hosts.

In order to minimize (potentially redundant) maintenance efforts, it may be considered to use the same type of deployment scripts/manifests on all levels.

Some Options:

1. Reuse K8s manifests at "low level" infrastructure hosts by using some slim K8s distribution there. In order to have as few moving parts as possible, features like dynamic persistent volume provisioning should be disabled. Maybe it is even viable to opt out of Pod networking and kube-dns. Such "low level" cluster is already required for the use of Gardener, AFAIK.
2. Reuse K8s manifests at "low level" infrastructure hosts by using podman play kube. AFAIK, this feature is only suitable for pretty simple use cases.