Kubernetes CI CD Pipelines and GitOps
Getting code from a developer's laptop into a running Kubernetes cluster involves many steps: building a container image, running tests, pushing the image to a registry, and updating the cluster. Doing this manually is slow and error-prone. CI/CD pipelines automate this process. GitOps takes it further by making Git the single source of truth for what should run in the cluster — and automatically reconciling the cluster to match.
What CI/CD Means in Kubernetes Context
Continuous Integration (CI)
CI is the automated process of building and testing code every time a developer pushes a change. In Kubernetes workflows, CI typically ends with a container image tagged and pushed to a container registry.
Continuous Delivery (CD)
CD takes the artifact produced by CI and deploys it to a Kubernetes cluster. This can be as simple as running helm upgrade or kubectl apply with the new image tag, or as sophisticated as a progressive canary rollout managed by a GitOps operator.
┌──────────────────────────────────────────────────────────────┐ │ CI/CD Pipeline Overview │ │ │ │ Developer CI System CD System Cluster │ │ ───────── ───────── ───────── ─────── │ │ git push ──► build image ──► update ──► new │ │ run tests manifests Pods │ │ push image apply to running │ │ to registry cluster │ └──────────────────────────────────────────────────────────────┘
A Typical Kubernetes CI Pipeline
The CI side focuses on producing a reliable, tested container image. Here is what a standard pipeline does:
Stage 1: Code Quality ├── Lint code (language-specific linter) └── Static analysis (SAST) Stage 2: Test ├── Unit tests └── Integration tests Stage 3: Build ├── docker build -t myapp:$GIT_COMMIT . └── Scan image for CVEs (Trivy, Grype) Stage 4: Push └── docker push registry/myapp:$GIT_COMMIT Stage 5: Update Manifest └── Update image tag in Kubernetes YAML or Helm values
The image tag should be the Git commit SHA, not latest. Using commit SHAs makes every image uniquely traceable to the exact code it contains. Tagging images as latest breaks rollbacks because you cannot tell which version of the code latest pointed to at any given time.
Example: GitHub Actions Pipeline
name: Build and Deploy
on:
push:
branches: [main]
env:
IMAGE: ghcr.io/myorg/my-app
jobs:
build-and-push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t $IMAGE:${{ github.sha }} .
- name: Scan for vulnerabilities
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.IMAGE }}:${{ github.sha }}
exit-code: 1 # Fail the pipeline on HIGH/CRITICAL CVEs
severity: HIGH,CRITICAL
- name: Push image
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u $ --password-stdin
docker push $IMAGE:${{ github.sha }}
- name: Update Helm values
run: |
sed -i "s/tag:.*/tag: ${{ github.sha }}/" charts/my-app/values-prod.yaml
git config user.email "ci@myorg.com"
git config user.name "CI Bot"
git add charts/my-app/values-prod.yaml
git commit -m "Deploy: ${{ github.sha }}"
git push
The last step commits the new image tag back to the Git repository. This is the entry point for GitOps — the CD system watches this repository and reconciles the cluster to the new state.
GitOps: Git as the Source of Truth
Traditional CD pipelines push changes to the cluster from the CI system. The pipeline has cluster credentials and runs kubectl apply or helm upgrade directly. This works but creates several problems:
- Credentials for the cluster must be stored as CI secrets, creating a wide attack surface.
- If someone manually changes a resource in the cluster, the CI system does not know and the cluster drifts from what is in Git.
- There is no central, auditable record of what is deployed and when.
GitOps flips the direction. An operator inside the cluster watches a Git repository. When it detects a difference between what Git says should be running and what is actually running, it reconciles the cluster to match Git — automatically.
┌──────────────────────────────────────────────────────────────┐ │ GitOps Model │ │ │ │ Git Repository Cluster │ │ ────────────── ─────── │ │ │ │ charts/ │ │ values-prod.yaml ◄────── GitOps Operator │ │ image.tag: abc12 (ArgoCD or Flux) │ │ watches Git │ │ every 3 minutes │ │ │ │ Dev pushes tag: def34 ──► Operator detects diff │ │ Applies update to cluster │ │ Cluster matches Git again │ │ │ │ Manual kubectl edit ───► Operator reverts it │ │ (drift detected) Cluster matches Git again │ └──────────────────────────────────────────────────────────────┘
ArgoCD: GitOps for Kubernetes
ArgoCD is the most widely adopted GitOps tool for Kubernetes. It runs inside the cluster, watches one or more Git repositories, and continuously reconciles the cluster to the desired state defined in those repositories.
Installing ArgoCD
kubectl create namespace argocd kubectl apply -n argocd -f \ https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Creating an ArgoCD Application
An ArgoCD Application resource tells ArgoCD: watch this Git repo, at this path, and deploy its contents to this cluster namespace.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/k8s-manifests
targetRevision: main
path: apps/my-app/production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true # Delete resources removed from Git
selfHeal: true # Revert manual changes in the cluster
syncOptions:
- CreateNamespace=true
automated.selfHeal: true is the key setting that prevents drift. Any manual kubectl edit or kubectl delete in the production namespace gets automatically reverted to match what Git says.
The ArgoCD UI
ArgoCD ships with a web UI that shows every Application, its sync status (whether the cluster matches Git), and the health of each Kubernetes resource. You can manually trigger a sync, view the diff between Git and the live cluster, and roll back to a previous Git revision.
# Port-forward to access the ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Get the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 -d
Flux: The CNCF GitOps Toolkit
Flux is the other major GitOps tool, maintained by the CNCF. Unlike ArgoCD, Flux has no built-in UI — it is entirely configuration-driven through Kubernetes custom resources. Flux tends to appeal to teams that prefer infrastructure-as-code approaches without a graphical interface.
Flux Key Resources
- GitRepository — Defines which Git repo and branch to watch.
- Kustomization — Applies a path in the GitRepository to the cluster, with reconciliation interval and health checks.
- HelmRelease — Manages a Helm chart release, automatically upgrading when the chart version or values change in Git.
- ImageUpdateAutomation — Watches a container registry for new image tags and automatically updates the Git repository with the new tag, triggering a reconciliation.
The GitOps Repository Layout
Teams structure their GitOps repository in different ways. Two common patterns are app-of-apps and environment directories.
Environment Directory Pattern
k8s-manifests/
├── apps/
│ ├── my-api/
│ │ ├── base/ # Shared resources
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ │ ├── dev/
│ │ │ └── values.yaml # Dev overrides (1 replica, small resources)
│ │ └── prod/
│ │ └── values.yaml # Prod overrides (5 replicas, larger resources)
│ └── my-worker/
│ └── ...
└── infrastructure/
├── cert-manager/
├── ingress-nginx/
└── monitoring/
The infrastructure/ directory holds cluster-wide tools. The apps/ directory holds application workloads. ArgoCD or Flux Application resources point to specific paths for specific namespaces.
Image Promotion: Moving Changes Across Environments
┌──────────────────────────────────────────────────────────────┐ │ Image Promotion Pipeline │ │ │ │ 1. CI builds image → registry/app:abc123 │ │ │ │ 2. CI updates dev/values.yaml: │ │ image.tag: abc123 │ │ Git commit → ArgoCD deploys to dev │ │ │ │ 3. Integration tests pass against dev │ │ │ │ 4. PR: update staging/values.yaml: │ │ image.tag: abc123 │ │ Merge → ArgoCD deploys to staging │ │ │ │ 5. Manual approval / automated gate │ │ │ │ 6. PR: update prod/values.yaml: │ │ image.tag: abc123 │ │ Merge → ArgoCD deploys to production │ └──────────────────────────────────────────────────────────────┘
Each environment promotion is a Git commit or a merged pull request. The Git log becomes an audit trail of every deployment, who approved it, and what changed.
Progressive Delivery: Canary and Blue-Green
A standard Kubernetes Deployment rolls out changes by gradually replacing old Pods with new ones. For higher confidence, progressive delivery tools like Argo Rollouts or Flagger provide more controlled strategies.
Canary Rollout
Send a small percentage of traffic to the new version. Watch error rates and latency. Gradually increase traffic to the new version. Automatically roll back if metrics degrade.
# Argo Rollouts canary example
spec:
strategy:
canary:
steps:
- setWeight: 10 # 10% of traffic to new version
- pause: {duration: 5m}
- setWeight: 50 # 50% if no issues
- pause: {duration: 5m}
- setWeight: 100 # Full rollout
analysis:
templates:
- templateName: success-rate
startingStep: 1
Blue-Green Deployment
Run the new version alongside the old version. Switch all traffic instantly once the new version passes health checks. The old version stays running briefly so you can switch back immediately if needed.
Secrets in GitOps Pipelines
Storing secrets in Git is a well-known anti-pattern — even in private repositories. Two tools solve this for GitOps workflows:
Sealed Secrets
Sealed Secrets encrypts a Kubernetes Secret using a public key. The encrypted SealedSecret resource is safe to commit to Git. The Sealed Secrets controller in the cluster holds the private key and decrypts it back into a regular Secret.
# Encrypt a secret kubectl create secret generic db-password \ --from-literal=password=mysecret \ --dry-run=client -o yaml | kubeseal -o yaml > sealed-secret.yaml # Commit sealed-secret.yaml to Git safely git add sealed-secret.yaml && git commit -m "Add sealed DB secret"
External Secrets Operator
External Secrets Operator connects the cluster to external secret stores like AWS Secrets Manager, HashiCorp Vault, or Google Secret Manager. You commit an ExternalSecret resource to Git that describes which secret to fetch. The operator fetches the actual secret value from the vault at runtime and creates a Kubernetes Secret.
Policy Enforcement: Preventing Insecure Deployments
GitOps ensures the cluster matches Git, but if insecure YAML gets into Git, it gets deployed. Admission controllers like Kyverno or OPA Gatekeeper act as gatekeepers, rejecting resources at the Kubernetes API level before they are applied.
# Kyverno policy: require all Pods to have resource limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-resource-limits
spec:
validationFailureAction: Enforce
rules:
- name: check-container-limits
match:
resources:
kinds: [Pod]
validate:
message: "All containers must have resource limits."
pattern:
spec:
containers:
- resources:
limits:
memory: "?*"
cpu: "?*"
This policy blocks any Pod without CPU and memory limits. Combined with GitOps, this means CI can catch the issue before it reaches the cluster, and the cluster itself refuses to apply it if CI misses it.
Key Points
- CI pipelines build, test, scan, and push container images; CD deploys those images to Kubernetes.
- Tag images with the Git commit SHA, not
latest, to make every deployment traceable and rollback-safe. - GitOps makes Git the single source of truth — a cluster operator continuously reconciles the cluster to match what Git defines.
- ArgoCD provides a UI-driven GitOps experience with automated sync and self-healing; Flux is a configuration-driven alternative.
- Promote image tags across environments through Git commits or pull requests, creating a full deployment audit trail.
- Use Sealed Secrets or External Secrets Operator to handle secrets in GitOps pipelines without storing plaintext credentials in Git.
- Argo Rollouts or Flagger enable canary and blue-green deployments for lower-risk production rollouts.
- Admission controllers like Kyverno enforce security and compliance policies at the cluster level, complementing GitOps pipelines.