pa-08 — The Hitchhiker's Guide to GitOps

Companion to CONCEPTS.md, with the runnable reconciler in src/go/gitops/. This is pa-07's plan/apply turned into a continuous, self-healing control loop — the ArgoCD/Flux model.

bash scripts/verify.sh runs the loop end to end:

initial sync (sync-wave order):  apply crd -> apply config -> apply workload
manual drift on 'config':        healed=[config] (reverted to "v1")
git removes 'config':            pruned=[config]; workload updated to "v2"
progressive delivery:            healthy -> v2;  unhealthy -> v1 rolledBack=true

1. The reconcile loop (reconcile.go)

Reconcile(desired, live, prune) is the loop body, meant to run continuously. It does three things, and the second is the GitOps superpower:

  • Sync — create missing resources, update changed ones (TestSyncCreates, TestSyncUpdatesOnGitChange).
  • Self-heal — the "update changed" branch also catches manual drift: TestSelfHealRevertsManualDrift hand-edits svc to HACKED and the next reconcile reverts it to git's v1. Sync and self-heal are the same mechanism; running it continuously is what makes the cluster unable to drift from git for long.
  • Prune — delete what git dropped (TestPruneDeletesRemoved), but only when enabled (TestNoPrunePreservesExtra) because prune is destructive.

TestReconcileIsIdempotent confirms a converged pass is a no-op — the property that makes running this every few seconds safe, shared with pa-07, gw-08, and gw-10.

2. Sync waves (reconcile.go)

TestSyncWaveOrdering applies crd (wave 0) → config (wave 1) → deploy (wave 2) regardless of declaration order. This is pa-07's dependency ordering as explicit phases — CRDs before the workloads that use them, namespaces before their contents — the same declaration-before-use concern as ADS in xDS (gw-08).

3. Pull, not push (the architecture decision)

The deep point isn't in the code, it's in the direction. A Jenkins pipeline pushes (kubectl apply from CI, with cluster credentials, once). GitOps pulls: an agent in the cluster watches git and converges. Consequences an architect cares about:

  • No drift survives — self-heal continuously corrects (vs push, where the cluster drifts freely between deploys).
  • No cluster creds in CI — the cluster reaches out; you don't expose it to the pipeline.
  • Git is change control + audit + rollback — every change is a reviewed PR (pa-10), every state a commit, rollback is git revert.
  • It scales — one reconciler per cluster handles thousands of apps.

4. Progressive delivery + the off switch

PromoteOrRollback(current, candidate, healthy) is the SLO-gated cutover in miniature (TestProgressiveDeliveryGate): healthy → promote; unhealthy → keep current (instant rollback, the old version never left). The full shadow → canary → ramp ladder with automated analysis is gw-12; GitOps makes the rollout state itself declarative and revertable.

The maturity detail the GUIDE insists on: automation needs a break-glass. Continuous self-heal will revert an on-call engineer's emergency manual fix. Real GitOps lets you pause reconcile for that window — and treats recurring drift as a signal that git is wrong.

5. The pattern, one more time

IaC (pa-07), GitOps (here), Kubernetes operators (gw-10), and xDS (gw-08) are the same level-triggered, idempotent, converge-to-desired-state loop. ArgoCD/Flux are literally Kubernetes operators for "the app in git." An architect who internalizes this designs every control plane the same way and reuses one operational playbook — the db-17 instinct ("drive the system to a replicated desired state") at the platform layer.

6. Hands-on

cd src/go
bash ../scripts/verify.sh
go run ./cmd/gitopssim

7. Exercises

  1. Continuous loop: run Reconcile on a ticker against a "live" map that a goroutine randomly drifts; show convergence within one tick.
  2. Break-glass: add a paused flag that suspends self-heal so an emergency manual change survives; log that drift exists while paused.
  3. Owner-ref-scoped prune: only prune resources this app owns, so a mis-scoped app can't delete another's resources.
  4. Wire SLO gating: replace the boolean healthy with a real burn-rate check (pa-09) and the canary ladder (gw-12).
  5. Multi-env: model dev/prod as separate desired sets and a promotion that's a git merge from dev→prod (the GitOps promotion pattern).