GitOps sounds simple: put your infrastructure in Git, let a controller sync it to your cluster. In practice, there are a dozen ways to get it wrong. Here’s what works.

The Core Principle

Git is the source of truth. Not the cluster. Not a dashboard. Not someone’s kubectl session.

Developers→inGgilt↑e→soCuorncteroller→Cluster

If the cluster state doesn’t match Git, the controller fixes it. If someone manually changes the cluster, the controller reverts it. This is the contract.

Pattern 1: App-of-Apps

Managing hundreds of applications individually doesn’t scale. Use a hierarchy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# apps/root.yaml - the "app of apps"
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/infrastructure
    path: apps
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# apps/production/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-production
spec:
  source:
    repoURL: https://github.com/org/api
    path: deploy/production
    targetRevision: v2.3.1  # pinned version
  destination:
    server: https://kubernetes.default.svc
    namespace: api

The root application watches the apps/ directory. Each file in that directory defines another application. Add a new service? Add a YAML file. Delete a service? Delete the file.

Pattern 2: Environment Promotion

Don’t copy manifests between environment directories. Use overlays:

d├│││├││└e───p───lob├├└s├└p├├└ya───t──r───/s───a──o───egd/dskikrukrreeunuecueeprsgsptspslvttlitlooioiooiuycmmcnmcrmeiia/iace.zz-z-enyaapap-tattatap.miititayloococtannhnhcm.....hlyyyyy.aaaaaymmmmmalllllml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# deploy/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# deploy/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - path: replica-patch.yaml
  - path: resource-patch.yaml

Promote by changing the image tag in base, not by copying files. The patches handle environment-specific differences.

Pattern 3: Image Update Automation

Manual image tag updates are tedious and error-prone. Automate them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Flux ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: api-image-update
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: infrastructure
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: flux@example.com
        name: Flux
      messageTemplate: 'chore: update {{.Changed.Name}} to {{.Changed.NewTag}}'
    push:
      branch: main
  update:
    path: ./deploy
    strategy: Setters

1
2
3
4
5

# In your deployment
spec:
  containers:
    - name: api
      image: registry.example.com/api:v1.2.3 # {"$imagepolicy": "flux-system:api"}

When a new image is pushed to the registry, Flux updates the tag in Git and commits. The cluster then syncs to the new commit. Full audit trail, no manual steps.

Pattern 4: Secrets Management

Secrets shouldn’t live in Git unencrypted. Options:

Sealed Secrets — Encrypt secrets client-side, decrypt in-cluster:

1
2
3
4
5

# Encrypt
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# The sealed secret is safe to commit
git add sealed-secret.yaml

External Secrets Operator — Reference secrets from a vault:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: api-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/api/database
        property: url

The secret value stays in AWS Secrets Manager. Kubernetes gets a synced copy. Rotate in the vault, the cluster follows.

Pattern 5: Drift Detection Without Auto-Sync

Sometimes you want to know about drift without automatically fixing it:

1
2
3
4
5
6
7

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    automated:
      prune: false      # Don't auto-delete
      selfHeal: false   # Don't auto-fix drift

Configure alerts instead:

1
2
3
4
5
6
7
8

# Prometheus alert
- alert: ArgoCDAppOutOfSync
  expr: argocd_app_info{sync_status!="Synced"} == 1
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Application {{ $labels.name }} is out of sync"

This is useful for stateful applications where automatic reconciliation could cause outages.

Pattern 6: PR Preview Environments

Every pull request gets its own environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# .github/workflows/preview.yaml
name: Preview Environment
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy-preview:
    runs-on: ubuntu-latest
    steps:
      - name: Create preview namespace
        run: |
          cat <<EOF | kubectl apply -f -
          apiVersion: v1
          kind: Namespace
          metadata:
            name: preview-${{ github.event.number }}
            labels:
              preview: "true"
              pr: "${{ github.event.number }}"
          EOF
      
      - name: Deploy to preview
        run: |
          kustomize build deploy/preview | \
            sed "s/NAMESPACE/preview-${{ github.event.number }}/g" | \
            kubectl apply -f -
      
      - name: Comment PR with URL
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: '🚀 Preview: https://pr-${{ github.event.number }}.preview.example.com'
            })

Clean up when the PR closes:

1
2
3
4
5
6
7
8
9

on:
  pull_request:
    types: [closed]

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - run: kubectl delete namespace preview-${{ github.event.number }}

Anti-Pattern: Manual Cluster Access

The moment someone runs kubectl apply directly against production, you’ve broken GitOps. The cluster state no longer matches Git.

Enforce it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# OPA Gatekeeper constraint
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockManualDeploys
metadata:
  name: block-manual-deploys
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
    namespaces: ["production"]
  parameters:
    allowedServiceAccounts:
      - "system:serviceaccount:argocd:argocd-application-controller"

Only the GitOps controller can modify production. Everyone else gets denied.

The Workflow

  1. Developer opens PR with infrastructure changes
  2. CI runs kustomize build and kubectl diff for review
  3. Reviewer approves based on the diff
  4. PR merges to main
  5. GitOps controller detects new commit
  6. Controller applies changes to cluster
  7. Controller reports sync status back to Git

No SSH keys to clusters. No kubeconfig files on laptops. No “let me just fix this real quick.” Everything through Git.

Getting Started

If you’re new to GitOps:

  1. Start with one non-critical app — Get the workflow right before scaling
  2. Use automated sync cautiously — Manual sync first, then enable automation
  3. Invest in observability — You need to see what the controller is doing
  4. Document your directory structure — Future you will thank present you

GitOps isn’t about the tools. It’s about the discipline of treating infrastructure changes like code changes: reviewed, versioned, and auditable.