pa-08 — GitOps & Progressive Delivery

The Apple JD lists "DevOps and CI/CD methodologies… such as ArgoCD, Flux, or Jenkins" and "familiar with GitOps workflows and progressive delivery practices." GitOps is a specific, powerful idea: git is the single source of truth for the desired state, and a reconciler continuously makes the running system match it — pulling, not pushing. It is pa-07's plan/apply turned into a continuous control loop with self-heal and prune.

You build the reconciler: sync (create/update), prune (delete what git removed), self-heal (revert manual drift), sync-wave ordering, drift detection, and an SLO-gated promotion.

1. What is it?

GitOps = declarative desired state in git + an agent that continuously reconciles the live system to it:

   git repo (desired state) ──pull──▶  [Reconciler]  ──converge──▶  cluster (live)
        ▲ commits = the only way to change prod        │
        │                                              ├─ SYNC: apply created/changed
   PR review = change control                          ├─ PRUNE: delete what git dropped
                                                       └─ SELF-HEAL: revert manual drift

Vs imperative push CD (Jenkins runs kubectl apply from a pipeline), GitOps pulls: the agent in the cluster watches git and converges, so the cluster can't drift from git for long (it self-heals), and the audit trail is the git history.

Progressive delivery layers safe rollout on top: canary / blue-green / SLO-gated promotion with automatic rollback (the full ladder is gw-12). GitOps makes the rollout itself declarative ("90% v1, 10% v2 in git").

2. Why does it matter?

• It closes the drift loop pa-07 left open. IaC detects drift; GitOps continuously corrects it. The cluster converges to git within seconds of any change — a hand-edit at 3am is reverted automatically (or surfaced). That's a fundamentally more reliable operational model.

• Git becomes change control + audit + rollback. Every change is a reviewed PR (a testing strategy, pa-10); every state is a commit; rollback is git revert. No "what's actually deployed?" mystery, no unaudited kubectl edits.

• Pull beats push for security and scale. The cluster pulls from git; you don't hand CI cluster-admin credentials or open the cluster to the pipeline. One reconciler per cluster scales to thousands of apps.

• It's the same reconcile loop, again. GitOps (here), IaC (pa-07), Kubernetes operators (gw-10), and xDS (gw-08) are all desired-vs-actual convergence. An architect who names that pattern designs control planes the same way every time — and it's db-17's "drive replicas to a desired state" instinct.

3. How does it work?

The reconcile loop

Reconcile(desired, live, prune) is the loop body, run continuously:

• Sync: a desired resource missing from live → create; present but different → update.
• Self-heal: that "present but different" case also catches manual drift — if someone hand-edited the cluster, live differs from git, so reconcile reverts it. Sync and self-heal are the same mechanism; the power is running it continuously.
• Prune: a live resource git no longer declares → delete (if pruning is enabled — a safety toggle, since prune is destructive).

Sync waves (ordering)

Resources carry a wave; the reconciler applies low waves first (CRDs before the workloads that use them; namespaces before resources in them). This is pa-07's dependency ordering expressed as explicit phases — the same "declaration before use" concern as ADS in xDS (gw-08).

Drift detection and idempotency

Diff(desired, live, prune) reports out-of-sync resources (missing, drifted, or extra). A converged reconcile is a no-op (idempotent) — the property shared by every reconcile loop in the book, and what makes running it every few seconds safe.

Progressive delivery + rollback

PromoteOrRollback(current, candidate, healthy) is the decision in miniature: promote the candidate only if healthy (SLO-gated), else keep the current version live (instant rollback — the old version never left). The full shadow → canary → ramp ladder with automated analysis is gw-12; GitOps makes the rollout state itself declarative and revertable.

4. Core terminology

5. Mental models

• GitOps is a thermostat wired to git. You set the target in git (the dial); the agent continuously drives the room (cluster) to it. Open a window (manual drift) and the thermostat works to close the gap. Push CD is "manually adjust the heater once and hope nobody touches it."

• Git is the system's single source of truth, including 'undo.' What should be running is whatever's committed; deploying is git push, rolling back is git revert, and the audit log is git log. The cluster is a derived, disposable projection of git.

• Prune is a chainsaw — useful, dangerous, toggle-guarded. "Delete whatever git doesn't mention" cleans up beautifully and can wipe a resource you forgot to commit. That's why prune is opt-in and sync-waves/owner-refs scope it.

• Self-heal vs break-glass. Continuous self-heal is great until an on-call engineer makes an emergency manual fix — which self-heal then reverts. Mature GitOps has a break-glass (pause reconcile) for exactly that. Automation must have an off switch.

6. Common misconceptions

• "GitOps is just CI/CD with git." The distinction is pull + continuous reconcile + self-heal: the cluster actively converges to git and corrects drift, rather than a pipeline pushing once and walking away. That continuous-convergence property is the whole point.

• "Prune is safe to leave on everywhere." Prune deletes anything not in git; an un-committed resource or a mis-scoped app can cause data loss. Enable deliberately, scope with owner references, and review prune diffs.

• "Self-heal means we never have incidents." It reverts config drift; it can also revert a human's emergency fix. You need a break-glass and to treat persistent drift as a signal (someone keeps fixing something git gets wrong).

• "GitOps replaces progressive delivery." They compose: GitOps is how the desired state is delivered; progressive delivery (gw-12) is how cautiously you shift traffic to a new version with rollback.

• "One giant git repo / app for everything." Blast radius: a bad commit syncs everywhere at once. Structure by app/environment, use sync waves and progressive delivery, and stage changes — the same migration discipline as gw-12.

7. Interview talking points

• "What is GitOps and how is it different from a Jenkins pipeline?" Git as the single source of truth + an in-cluster reconciler that pulls and continuously converges (sync/prune/self-heal). Vs a pipeline that pushes once: GitOps self-corrects drift, doesn't need cluster creds in CI, and makes git the audit log + rollback (git revert).

• "How does self-heal work and when is it dangerous?" The reconcile loop detects live ≠ git (drift) and reverts to git — same mechanism as sync, run continuously. Dangerous when it reverts an emergency manual fix; mitigate with a break-glass (pause) and by treating recurring drift as a bug in git.

• "How do you order dependent resources?" Sync waves (CRDs/namespaces before workloads) — pa-07's dependency ordering as explicit phases. Same declaration-before-use concern as xDS ADS (gw-08).

• "GitOps + progressive delivery together?" GitOps delivers the declared state; progressive delivery (gw-12) gates the version cutover with canary/SLO analysis and auto-rollback. The rollout itself is declared in git and revertable.

• "It's the same loop as Kubernetes/Terraform/xDS — why does that matter?" Desired-vs-actual convergence is the universal control paradigm. Recognizing it means you design control planes consistently (idempotent, level-triggered, drift-correcting) and reuse the same operational playbook (db-17's instinct at the platform layer).

8. Connections to other labs

• pa-07 (IaC) — GitOps is plan/apply as a continuous loop with self-heal + prune; both are the reconcile paradigm.
• gw-10 / gw-08 (operators / xDS) — the same loop hosted in Kubernetes / a control plane; ArgoCD/Flux are themselves operators.
• gw-12 (progressive delivery) — the full shadow→canary→ramp ladder this lab's promotion gate references.
• pa-09 (SLOs) — promotion gates and rollback triggers are SLO decisions; pa-10 — PR review of git changes is a testing strategy / consensus mechanism.
• db-17 (Raft) — converge to a replicated desired state is the consensus instinct underlying GitOps.