Docs / CI/CD and GitOps

Kubernetes, Argo CD and Flux

changeward k8s gate plans a set of rendered manifests against the live cluster, using a server-side dry run with managed fields and status stripped. Policy then decides. The command exits 0 only if that exact plan is approved.

You can run it in three places:

  • as a PR check on your GitOps repository;
  • as an Argo CD PreSync hook, so a sync waits until its plan is approved;
  • locally before a manual rollout.
$ changeward k8s gate --manifests apps/shop --target cluster
skipped 1 hook resource(s) (Argo CD / Helm hooks are not desired state)
plan p_f88610636592 · cluster (prod) · reversible · 1 objects · server-side dry run
~ shop/ConfigMap/shop-config (update)
     data.feature_new_cart: "false" → "true"
change c_9dd33d8939b8 (#3) — awaiting-approval
policy: approval required (out-of-band; approvers: group:sre): production Kubernetes syncs need SRE approval
changeward: gate: approval required; approve with `changeward approve p_f88610636592` (or Slack/web), then retry the sync
  • Approvals bind to the plan hash.
  • A retried gate with the same manifests and the same live state reuses the decision, so it passes once approved.
  • A different commit, or a cluster that drifted, needs a new approval.

Pass --executor URL (token in $CHANGEWARD_EXECUTOR_TOKEN) to run the gate on a shared executor. The executor then holds the cluster credentials and the approvals. --out plan.txt writes the diff for a PR comment.

Target

changeward connect cluster --adapter kubernetes --env prod --option context=prod-eu --option namespace=shop

Options: context, kubeconfig, namespace, kubectl (binary path) and rollout_timeout. Inside a cluster the executor uses its service account. The Helm chart can create one with namespaced or cluster-wide RBAC (rbac.kubernetes.enabled).

Manifests must be rendered YAML. For Helm or Kustomize, render first (helm template, kustomize build).

Argo CD (PreSync hook)

Verified with Argo CD v3.5.5 in kind, using the in-cluster executor and a private GitLab repository:

  1. A reversible create was auto-approved.
  2. A feature-flag change failed the hook with the exact diff.
  3. An SRE ran changeward approve --executor and saw the plan before approving.
  4. Argo's retry passed and the change was live.
  5. The gate's own token couldn't approve.

Add integrations/argocd/presync-gate.yaml to the app's manifests:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: changeward-gate-
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: gate
          image: ghcr.io/srawlin/changeward:latest
          env:
            - { name: REPO_URL, value: "https://github.com/your-org/gitops.git" }
            - { name: REVISION, value: "main" }
            - { name: MANIFESTS_PATH, value: "apps/shop" }
            - { name: CHANGEWARD_EXECUTOR_URL, value: "http://changeward-executor.changeward.svc:8443" }
            - name: CHANGEWARD_EXECUTOR_TOKEN
              valueFrom: { secretKeyRef: { name: changeward-gate-token, key: token } }
          command: ["sh", "-c"]
          args:
            - git clone --quiet --depth 1 --branch "$REVISION" "$REPO_URL" /tmp/repo &&
              changeward k8s gate --manifests "/tmp/repo/$MANIFESTS_PATH" --target cluster

Then:

  • Retries. Give the Application syncPolicy.retry with enough attempts and backoff for a human to approve.

  • Tokens. Create two executor tokens:

    • one for the gate (--roles propose), stored in the changeward-gate-token Secret;
    • one per approver (--roles approve).

    Approve with:

    CHANGEWARD_EXECUTOR_URL=https://changeward.internal:8443 CHANGEWARD_EXECUTOR_TOKEN=<approver token> \
      changeward approve c_9dd33d8939b8
    

    The approval can also come from Slack or the web view.

  • Private repositories. Clone with a read-only token from the Secret: https://oauth2:${GIT_TOKEN}@${REPO_URL#https://}.

  • Hook resources. The gate drops Argo CD and Helm hook resources from the plan, including its own Job.

Flux

Flux has no pre-sync hook, so gate where changes enter Git:

  1. GitOps PRs. Run changeward k8s gate --manifests <rendered> --target <cluster> --executor … in CI, and make it a required check. The plan is computed against the live cluster.
  2. Incidents. flux suspend kustomization <name>, then change through Changeward, then changeward reconcile --k8s-dir … --pr to bring Git in line, then flux resume.

The adapter flags objects owned by Argo CD or Flux in direct-lane plans: "a direct change will be reverted on the next sync". changeward drift --max-age 24h exits 3 if direct changes stay unreconciled.

GitHub Actions example (PR check)

  k8s-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: kustomize build overlays/prod > rendered.yaml
      - run: changeward k8s gate --manifests rendered.yaml --target cluster --executor "$CHANGEWARD_EXECUTOR_URL" --out plan.txt
        env:
          CHANGEWARD_EXECUTOR_URL: ${{ vars.CHANGEWARD_EXECUTOR_URL }}
          CHANGEWARD_EXECUTOR_TOKEN: ${{ secrets.CHANGEWARD_GATE_TOKEN }}

The executor must be reachable from the runner, e.g. through a self-hosted runner in the cluster's network.

Edit this page on GitHub · synced from 89c04c9