gitops-patterns icon indicating copy to clipboard operation
gitops-patterns copied to clipboard

verified publisher icon cloudogu
→

Metadata

Collection of patterns, examples and resources for GitOps process design, GitOps repository structures, etc

cloudogu/gitops-patterns

Intro

This repo collects some patterns (AKA strategies, models, approaches, best practices, etc.) about GitOps process design (AKA GitOps repository structures, etc.) as well as links to talks, articles and example repos.

It originated in schnatterer's talk about the "perfect" GitOps process, our experience in consulting, and our GitOps trainings for both Flux and ArgoCD.

You can read more about the patterns in this series of articles.

Discuss it on myCloudogu

PRs welcome!

Contents

  • Articles
  • Pattern categories
  • GitOps Patterns
    • Operator deployment
      • Hub and Spoke
      • Instance per Cluster
      • Instance per Namspace
      • Split-Instance
    • Repository structure
      • Monorepo
      • Repo per Team
      • Repo per Application (can be implemented via Repo Separation, Config Replication, Repo Pointer or Config Split)
      • Repo per Environment
    • Promotion
      • Environments
        • Folder/Directory per environment
        • Repo per environment
        • Branch per environment
        • Preview environments
      • Implementation
        • Configuration Management, includes Rendered Manifests and Umbrella Chart
        • Global Environments vs Environment per App
        • Config update
    • Wiring
      • Bootstrapping
      • Linking, includes Nesting (e.g. App of Apps), Templating
  • Public GitOps Repo Examples
    • GitOps Playground
    • ArgoCD autopilot
    • Flux Monorepo
    • Flux repo per team/tenant
    • 📕 Path to GitOps examples
    • Environment variations
    • Multiple AppSets
    • Others
  • Synonyms
  • References

Articles

article icon Part 1️⃣: GitOps Chasm + categories
article icon Part 2️⃣: Deployment patterns
article icon Part 3️⃣: Repo patterns
article icon Part 4️⃣: Promotion patterns
article icon Part 5️⃣: Wiring patterns + conclusion
article icon Part 6️⃣: Example Repositories

Pattern categories

Let's group GitOps patterns into separate categories to make them easier to grasp.
See also part 1️⃣ of the article series.

  • Operator deployment: GitOps operators ↔ Clusters/Namespaces
  • Repository: How many repos?
  • Promotion:
    • Environments: How to model environments/stages?
    • Implementation: How to implement the actual process of promotion?
  • Wiring: Bootstrapping operator, linking repos and folders

GitOps Patterns

Operator deployment

See also part 2️⃣ of the article series.

  • Hub and Spoke [^1][^2] 1 Operator : n Clusters
    Synonyms: Management Cluster [^18] - sounds familiar, e.g. from Cluster API, but it requires a cluster. In SaaS/PaaS constellations, the central instance may not even be operated in a cluster. Therefore, the term "Hub and Spoke" seems more fitting to us, especially since it is also well known from computer networks or logistics.
    Hube and spoke
  • Instance Per Cluster [^18] : 1 Operator : 1 Cluster
    Synonyms: Standalone [^1] - could apply to both "Instance per Cluster" and "Instance per Namespace"; does not define the relationship to clusters and namespaces precisely enough, as not every operator supports both deployment patterns.
    Instance Per Cluster
  • Instance per Namespace: n Operators : 1 Cluster
    Synonyms: Namespaced [^3]
    Instance per Namespace

More Patterns:

  • Split-Instance[^1]: 1 Operator : n Clusters; components split between management and target clusters
  • Hybrids of Hub and Spoke and Instance per Cluster patterns: "Instance per Logical Group" and "Argo Managing more Argos".[^18]

Repository structure

See also part 3️⃣ of the article series.

  • Monorepo [^2][^4]
    Opposite: polyrepo [^2]
  • Repo per Team [^4]
    More generalized: Team ➡️ Tenant[^4]
  • Repo per Application [^2][^4]
    Implementations:
    • Repo separation [^19]
      Keep code in app repo, config in config repo
    • Config replication [^3]
      Keep config in app repo and have CI server replicate it to the config repo
    • Repo pointer [^6][^3][^4]
      Keep the whole config in app repo and add a pointer from config repo (e.g. Argo CD Application or a Flux GitRepository+Kustomization), avoiding redundancy
    • Config Split [^20]
      Keep parts of the config in app repo (e.g. helm chart or kustomize base), and rest in config repo (e.g. value.yamls or overlays for different envs).
      Then have the config repo point to the app repo (e.g. via ArgoCD Application or Flux Kustomization).

      Alternative implementations: have CI server push chart to helm registry

      Or use push helm chart to OCI registry

      Or use any config management tool (e.g. helm template, kustomize build, kubectl kustomize, jsonnet, cuelang, timoni, etc. ) on the CI server for pushing the final manifests as OCI artifacts[^22] to the registry (Rendered Manifests Pattern[^23]).
      Then have the config repo point to the OCI artifact (e.g. via Flux Kustomization).
      This way, the OCI registry functions as a "GitOps Cache"[^21]:
      The operator only needs to pull the artifacts instead of rendering/overlaying the config from different sources.
  • Repo per environment [^4]
    Synonym: Environment per repository[^5], Repo per Stage

Promotion

See also part 4️⃣ of the article series.

We understand the term "promotion" as the process of deploying applications to different environments.
Sometimes, the term "promotion" is prefixed with other words: Release/Application/Environment/Workload/Change.

For promotion, we see different sets of patterns:

  • one regarding the modelling of environments and
  • one regarding the implementation of the actual process of promotion.

Environments

  • Folder/Directory per environment
    Synonym: Environment per folder [^5]
  • Repo per environment [^4]
    Synonym: Environment per repo [^5]
  • Branch per environment
    Synonym: Environment per branch [^5]
    Often discouraged [^6] or declared an anti-pattern[^7][^14], but can work [^13].
  • Preview environments [^9] [^10] [^11]
    Synonyms: ephemeral, dynamic, pull request[^8], test, temporary[^9]
    Beyond the GitOps world also known as "Preview Deployments"[^15] and "Deploy Previews"[^16]

Implementation

  • Configuration Management
    • Synonyms:
      • General: Templating, Patching, Overlay, Rendering, Hydrating
      • templates = dry; rendered = hydrated [^26]
    • Where to render the manifests?
      • On the CI Server (Rendered Manifests Pattern[^23])
        This pattern also fits nicely with Flux's OCI artifacts feature (see Config split).
      • On the GitOps Operator (GitOps Operator Rendering or GitOps Controller Rendering[^25])
    • Tools:
      • Kustomize
        • Plainkustomization.yaml - operator-agnostic (GitOps Operator Rendering on Argo CD and Flux)
        • ≠ Flux CRD Kustomization (GitOps Operator Rendering)
        • kustomize build / kubectl kustomize via CI server (Rendered Manifests)
      • Helm
        • CRD such as HelmRelease (GitOps Operator Rendering on Flux) or Application (GitOps Operator Rendering on ArgoCD)
        • Umbrella Chart[^12] (common on Argo CD, possible but unusual on Flux)
        • helm template on CI server (Rendered Manifests)
      • Others, e.g. Jsonnet, CueLang, Timoni
        • Rendered Manifests Pattern
        • Argo CD Config Management Plugins[^24] (built-in for JSonnet)
  • Global Environments vs Environment per App[^3]
    Global Envs Env per app
  • Config update
    Who updates image (version) in GitOps repo, creates branch and PR?
    • Manual: Human pushes branch and create PR 🥵
    • CI Server: Build job pushes branch, creates PR
    • Image Updater: Operator pushes branch, create PR manually
    • Dependency Bot: Bot pushes branch, creates PR

Wiring

See also part 5️⃣ of the article series.

  • Bootstrapping , e.g. using kubectl or operator-specific CLI such as flux or argocd-autopilot
  • Linking e.g. using Operator-specific CRDs such as Kustomization (Flux) or Application (ArgoCD)
    • Nesting, e.g. App of Apps pattern [^17] (ArgoCD)
    • Templating, e.g. implemented using ApplicationSets

Public GitOps Repo Examples

See also [^3].

See also part 6️⃣ of the article series.

The diagrams have been created with tree2svg.

GitOps Playground

cloudogu/gitops-playground

  • Repo pattern: Per team mixed with per app (implemented via Config Replication)
  • Operator pattern: Instance per Cluster (Hub and Spoke also possible)
  • Operator: ArgoCD (Flux)
  • Boostrapping: Helm, kubectl
  • Linking: ArgoCD Application
  • Features:
    • Operate ArgoCD with GitOps
    • Solution for cross-cutting infra (cluster resources)
    • Multi-tenancy: Single operator manages multiple tenants on one cluster using namespace environments (multi-cluster possible)
    • Mixed repo patterns
    • Config Update and Config Replication via CI server (Rendered Manifest using Helm)
    • Env per app Pattern

ArgoCD autopilot

argoproj-labs/argocd-autopilot

  • Repo pattern: Monorepo
  • Operator pattern: Instance per Cluster / Hub and Spoke
  • Operator: ArgoCD
  • Boostrapping: argocd-autopilot
  • Linking: kustomization.yaml, ArgoCD Application, ApplicationSet
  • Features:
    • Create structure and YAML via CLI
    • Operate ArgoCD with GitOps
    • Solution for cross-cutting infra (cluster resources)
    • Env per app Pattern

Flux Monorepo

fluxcd/flux2-kustomize-helm-example

  • Repo pattern: Monorepo
  • Operator pattern: Instance per Cluster
  • Operator: Flux
  • Boostrapping: flux CLI
  • Linking: kustomization.yaml, Flux Kustomization
  • Features:
    • Operate Flux with GitOps
    • Solution for cross-cutting infra (cluster resources)

Flux repo per team/tenant

fluxcd/flux2-multi-tenancy

  • Repo pattern: Repo per team/tenant
  • Operator pattern: Instance per Cluster
  • Operator: Flux
  • Boostrapping: flux CLI
  • Linking: kustomization.yaml, Flux Kustomization
  • Features:
    • Operate Flux with GitOps
    • Solution for cross-cutting infra (cluster resources)
    • Multi-tenancy: One operator per cluster environment manages multiple tenants

📕 Path to GitOps examples

christianh814/example-kubernetes-go-repo

christianh814/example-kubernetes-goflux-repo

christianh814/example-openshift-go-repo

  • Repo pattern: Monorepo
  • Operator pattern: Instance per Cluster
  • Operator: Argo CD, Flux
  • Boostrapping: kubectl
  • Linking: kustomization.yaml,
    • ArgoCD Application, ApplicationSet /
    • Flux Kustomization
  • Features:
    • Cross-cutting infra and app(s)
    • ArgoCD and Flux examples
    • Env per app Pattern

Environment variations

kostis-codefresh/gitops-environment-promotion

  • Operator: ArgoCD (Flux)
  • Linking: kustomization.yaml
  • Features:
    • Env variants for single app
    • Promotion "via cp"
    • Env per app Pattern

Multiple AppSets

kostis-codefresh/many-appsets-demo

  • Operator: Argo CD
  • Linking: kustomization.yaml, Argo CD ApplicationSet, (Application)
  • Features:
    • Env variants for multiple apps with few appSets
    • Promotion "via cp"
    • Env per app Pattern

Others

Here are some other examples that we haven't had a chance to look at in more detail:

Synonyms

  • Patterns ≈ strategies, models, approaches, best practices, standards
  • GitOps process design ≈ GitOps repository structures,
  • GitOps Operator ≈ GitOps controller ≈ GitOps agent
  • Config Repo = GitOps repo, Infra repo, Payload repo
    Config repo example
  • App repo = Source code repo, Source repo
    App repo example
  • Environment = Stage
  • Folder = Directory
  • Templating ≈ Patching, Overlay, Rendering, Bundling, Packaging?

References

[^1]: Article A Comprehensive Overview of Argo CD Architectures – 2023 by Dan Garfield
[^2]: Article/Book How to set up your GitOps directory structure by Christian Hernandez
[^3]: Slides The perfect GitOps process: repos, folders, stages, patterns by Johannes Schnatterer
[^4]: Documentation Flux | Ways of structuring your repositories
[^5]: Lesson GitOps at Scale Lesson series - Git repository strategies by Codefresh (paywalled)
[^6]: Talk GitOps: Core Concepts & Ways of Structuring Your Repos by Pinky Ravi and Scott Rigby
[^7]: Article Stop Using Branches for Deploying to Different GitOps Environments by Kostis Kapelonis
[^8]: Video Environments Based On Pull Requests (PRs): Using Argo CD To Apply GitOps Principles On Previews by Viktor Farcic
[^9]: Article Creating Temporary Preview Environments Based On Pull Requests With Argo CD And Codefresh by Codefresh
[^10]: Talk GitOps Con Europe - Implementing Preview Environments with GitOps in Kubernetes by François Le Pape, Remazing
[^11]: Talk Preview Environments with ArgoCD by Brandon Phillips
[^12]: Documentation Helm | Chart Development Tips and Tricks
[^13]: Article Monitoring and Hardening the GitOps Delivery Pipeline with Flux by Florian Heubeck
[^14]: Article Git best practices: Workflows for GitOps deployments by Christian Hernandez
[^15]: Documentation Vercel: Preview Deployments
[^16]: Documentation Netlify: Deploy Previews
[^17]: Documentation ArgoCD: Cluster Bootstrapping - App Of Apps Pattern
[^18]: Talk Control Plane, Service, or Both? – Argo CD Multi-Cluster Architectures - Nicholas Morey, Akuity, Article How many do you need? - Argo CD Architectures Explained by Nicholas Morey
[^19]: Documentation Argo CD: Best Practices
[^20]: Discussion on LinkedIn Benjamin Ruland and Johannes Schnatterer
[^21]: Talk: Mastering GitOps 2023: Keynote: GitOps Emerging Developments and Predictions by Alexis Richardson
[^22]: Documentation Flux | OCI cheatsheet
[^23]: Talk: GitOpsCon EU 23: The Art of GitOps: Rendered Manifests by Christian Hernandez
[^24]: Documentation Argo CD | Config Management Plugins [^25]: Discussion on LinkedIn Christian Hernandez, Scott Rigby and Johannes Schnatterer
[^26]: Discussion on GitHub Michael Crenshaw

← Metadata

276
Stars
32
Forks
Watchers

Owner

verified publisher icon cloudogu

Metadata

Collection of patterns, examples and resources for GitOps process design, GitOps repository structures, etc

Back