GitOps Done Right: When Git Becomes Your Control Plane

GitOps sounds simple: Git is the source of truth, automation syncs it to reality. In practice, most teams get it wrong. Here’s how to get it right.

The Core Principle

GitOps isn’t “we use Git.” It’s a specific operational model:

1. Declarative: You describe what you want, not how to get there
2. Versioned: All changes go through Git (audit trail for free)
3. Automated: Software agents continuously reconcile desired vs actual state
4. Observable: You can always answer “what’s deployed where?”

The magic is in reconciliation. Traditional CI/CD pushes changes. GitOps pulls desired state and converges toward it. The system heals itself.

Push vs Pull: Why It Matters

Push model (traditional CI/CD):

Problems:

• CI needs cluster credentials (security risk)
• If apply fails mid-way, state is inconsistent
• No drift detection — manual changes go unnoticed
• “It worked in CI” doesn’t mean it’s still working

Pull model (GitOps):

Benefits:

• Cluster pulls from Git (no external credentials needed)
• Continuous reconciliation fixes drift
• Agent retries until desired state matches actual
• Single source of truth, always

The Tools That Work

Argo CD — The most popular choice. Declarative, UI-rich, handles complex deployments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/my-app-config
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Key settings:

• selfHeal: true — Reverts manual cluster changes
• prune: true — Removes resources deleted from Git
• CreateNamespace=true — Creates namespace if missing

Flux — Lighter weight, more composable. Good for multi-tenancy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# flux-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app-config
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: my-app
      namespace: my-app

Both work. Argo CD has better visualization. Flux is more Kubernetes-native. Pick one and commit.

Repository Structure That Scales

The biggest GitOps mistake: mixing application code with deployment config.

Wrong:

Right — separate repos:

Why separate?

• Different change velocity (code changes daily, infra config less often)
• Different access controls (devs vs platform team)
• Cleaner Git history per concern
• CI updates config repo with new image tags

The Image Update Problem

Your app builds a new image. How does it get deployed?

Option 1: CI updates the config repo

1
2
3
4
5
6
7

# .github/workflows/build.yaml
- name: Update image tag
  run: |
    cd my-app-config
    kustomize edit set image my-app=myregistry/my-app:${{ github.sha }}
    git commit -am "Update my-app to ${{ github.sha }}"
    git push

Simple, explicit, works. Downside: CI needs write access to config repo.

Option 2: Image automation (Flux)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: my-app
spec:
  imageRepositoryRef:
    name: my-app
  policy:
    semver:
      range: ">=1.0.0"
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: my-app
spec:
  sourceRef:
    kind: GitRepository
    name: my-app-config
  update:
    strategy: Setters
    path: ./overlays/production

Flux watches the registry, updates Git when new tags appear. Fully automated. Magic when it works, confusing when it doesn’t.

I prefer Option 1 for clarity. You can always see who pushed the change.

Secrets: The Hard Part

Secrets can’t live in Git. But GitOps wants everything in Git. Solutions:

Sealed Secrets — Encrypt secrets client-side, only the cluster can decrypt.

1
2
3
4
5

# Encrypt a secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Commit the sealed version
git add sealed-secret.yaml

Simple, but key rotation is manual.

External Secrets Operator — Syncs secrets from external stores (Vault, AWS Secrets Manager).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: my-app-secrets
  data:
    - secretKey: database-password
      remoteRef:
        key: secret/my-app
        property: db_password

Git contains references to secrets, not secrets themselves. Best of both worlds.

SOPS — Encrypts files in place, integrates with Flux.

1
2
3
4

# .sops.yaml
creation_rules:
  - path_regex: .*secrets.*\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/abc123

Decrypt on the fly during sync. Works well for smaller teams.

Multi-Cluster GitOps

One Git repo, many clusters. How?

Kustomize overlays:

Each overlay patches base config for its cluster. Argo CD ApplicationSets can generate Applications per cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: production
  template:
    metadata:
      name: 'my-app-{{name}}'
    spec:
      source:
        repoURL: https://github.com/myorg/config
        path: 'overlays/{{name}}'
      destination:
        server: '{{server}}'

One definition, deployed everywhere. Changes propagate automatically.

Progressive Delivery

GitOps + progressive rollouts = confidence.

Argo Rollouts integrates with Argo CD:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 100
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1

Git says “deploy version X.” Rollout controller says “carefully, with analysis.”

Common Mistakes

1. Not enabling pruning

Without prune: true, deleted resources in Git stay in the cluster forever. Enable it.

2. Manual cluster changes

Someone kubectl edits a deployment. GitOps reverts it. They do it again. Enable selfHeal and educate the team: Git is the only way.

3. Monolithic config repos

One massive repo for everything = slow syncs, merge conflicts, unclear ownership. Split by team or service boundary.

4. Ignoring sync status

“It’s in Git” doesn’t mean “it’s deployed.” Monitor sync status. Alert on failed syncs. Argo CD and Flux both expose metrics.

5. No promotion strategy

How do changes flow dev → staging → production? Manual PRs? Automated promotion? Define it explicitly.

Start Here

1. This week: Set up Argo CD or Flux in a dev cluster
2. Next week: Move one app’s config to a GitOps repo
3. This month: Add automated image updates
4. This quarter: Multi-cluster with progressive delivery

GitOps is a practice, not a product. The tooling helps, but the discipline — everything through Git, always reconciling — is what delivers the value.

The best infrastructure is the one you can recreate from a Git checkout. GitOps makes that the default, not the exception.