Staging Cluster

[JamWils]

I want to set up a multi-cluster environment where we can have both Weave GitOps Core and Weave GitOps Enterprise running. This will give us a development environment that has OIDC set up for development purposes enabling us to break out of the cycle of engineers just testing the software on their local machine (via single cluster) with “super power” privileges.

As a quality engineer or product owner, I always want to be able to work with the latest version of Weave GitOps Core and Enterprise that is sitting on main.

As a product owner, I want to be able to sign in as different users with varying permissions to test the app.

As product owner, I always want to be able to work with the latest official version of Weave GitOps Core and Enterprise.

Criteria

These are small machines since we don’t need to run huge workloads. The example below is GCP but AWS is fine too.

• 1 Google Cloud GKE Cluster (us-central)
• 1 Google Cloud GKE Cluster (eu-central)

GitHub Repository

We have a GitHub repository configuration repository that drives the automation of the 3 clusters. I would prefer that we start with just the two GKE clusters and work towards adding the EKS cluster later. We can run flux bootstrap on each cluster and have Weave GitOps Core and Enterprise installed.

Requirements

• We want the image automation controller to also be installed via Flux.
• We want semantic versioning automation set up for Core and Enterprise so we pull the latest release in. Automate image updates to Git | Flux (fluxcd.io)
• Sockshop is installed via a kustomization within the git repository and is added to both clusters.
• External access. We want to allow engineers to access both Weave GitOps Core and Enterprise. E.g. core.gitops-dev.com and enterprise.gitops-dev.com

OIDC Research

We need to figure out a low friction way to enable engineering, product, and quality to create users with different permissions so we can explore the application's functionality accurately. @yiannistri will be a good contact as he set up a similar environment while building IAM.

Stretch Goal

Every merge to main produces a new container image that would get uploaded to a separate container registry and not the default weave-gitops on GitHub. We have a separate kustomization in a namespace that is always able to run the latest image from main for both Core and Enterprise.

[SamLR]

WRT Autopilot Vs Standard GKE looks like it's worth going with a standard cluster:

• single node/single region small node/ standard cluster: $7.12pcm
• 0.5vCPU/512Mib deployment on autopilot: $92.91pcm

The bulk of the cost for autopilot comes from the fixed cluster costs but between the greater price and uncertainty as to how various features of gitops (especially enterprise) will interact with autopilot probably best to go with a standard cluster.

[SamLR]

Current plan is roughly 4 stages:

stage 1

• [X] gcloud project
• [X] repo for cluster config
• [x] terraform state bucket in GCS
• [x] google container registry for builds of v2
• [x] GHA to generate images in google container registry

stage 2

• [x] gke cluster
• [x] auto deploy flux
• [x] deploy gitops-ui etc.
• [x] flux image automation

See bottom for up to date planned next steps stage 3 ~- [ ] 2nd gke cluster~ ~- [ ] deploy EE (+ flux/gitops-ui/FIA)~

stage 4 ~- [ ] EKS cluster~ ~- [ ] CAPI?~

[SamLR]

terraform is living here: https://github.com/weaveworks/weave-gitops-clusters

[SamLR]

latest PR https://github.com/weaveworks/weave-gitops-clusters/pull/8

One final step to 'phase 2' -- adding an identity aware load balancer keyed to the weaveworks gsuite in front of the UI so that we can run it less securely and have it remain inaccessible unless you work for us.

[SamLR]

Revised plan

1. [x] Install sockshop dev and staging and podinfo on the cluster (AKA some sort of workload to manage)
2. [x] Set up DNS + HTTPS record to the single cluster. (AKA easy(er) access)
3. [x] Set up an Enterprise instance of it on the single cluster.
4. [ ] Whenever there is a merge to main we should be able to install the latest version of Enterprise.
5. [ ] Set up a second cluster with GCP

STRETCH:

1. [ ] Create dev users with scoped down permissions.

[SamLR]

going to skip installing sock-shop as the manifests for that are very old (using extensions/v1beta1 for deployments)

[SamLR]

going to look at limiting to cluster create/delete etc to me (& limited others)

all will be able to use enterprise ui for view

stretch - can we use flintlock to provision micro-clusters?

[SamLR]

[ ] metrics & logs

[SamLR]

[ ] metrics & logs

This is somewhat dealt with using google's integrated solutions

[JamWils]

I'm closing this for now since this work has moved over to another team.