Kubernetes DaemonSets and Jobs Running Special Workloads

Deployments and StatefulSets handle long-running services. Some workloads have different needs: run exactly one copy per node, or run once and stop when done. Kubernetes provides DaemonSets and Jobs for these cases.

DaemonSets: One Pod Per Node

A DaemonSet ensures that exactly one copy of a Pod runs on every node in the cluster (or a selected subset). When you add a new node, the DaemonSet automatically starts a Pod on it. When you remove a node, the Pod is garbage-collected.

Cluster with 4 nodes:

Node 1: [DaemonSet Pod]  ← 1 per node, always
Node 2: [DaemonSet Pod]
Node 3: [DaemonSet Pod]
Node 4: [DaemonSet Pod]

Add Node 5 → DaemonSet Pod automatically starts on Node 5
Remove Node 3 → DaemonSet Pod on Node 3 cleaned up

Common DaemonSet Use Cases

• Log collectors: Fluentd or Filebeat reads logs from every node and ships them to Elasticsearch or Splunk.
• Monitoring agents: Prometheus Node Exporter collects hardware and OS metrics from every node.
• Network plugins: CNI plugins like Calico and Cilium run as DaemonSets to manage per-node networking.
• Security agents: Host-based intrusion detection tools that inspect every node's activity.
• Storage drivers: CSI (Container Storage Interface) node plugins run as DaemonSets on each node.

Writing a DaemonSet YAML

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluentd:v1.16
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

The hostPath volume gives the container direct access to the node's log directory — something only DaemonSets and privileged workloads typically need. The toleration allows this DaemonSet to also run on control plane nodes.

Jobs: Run Once and Stop

A Job creates one or more Pods to complete a task. When all Pods finish successfully, the Job is done. Unlike Deployments, Kubernetes does not restart successful Job Pods — it leaves them in a Completed state for you to inspect the output.

Use Jobs for database migrations, report generation, data processing pipelines, one-time scripts, and batch computations.

Job creates Pod → Pod runs the task → Task completes
                                          ↓
                                   Pod status: Completed
                                   Job status: Complete

Writing a Job YAML

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migration
spec:
  completions: 1       # Run 1 successful completion
  parallelism: 1       # Run 1 Pod at a time
  backoffLimit: 3      # Retry up to 3 times on failure
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: migrate
        image: my-app:v2
        command: ["python", "manage.py", "migrate"]
        env:
        - name: DB_HOST
          value: "postgres.production.svc.cluster.local"

The Job runs the migration script. If it fails, Kubernetes retries up to 3 times. On success, the Pod stays in Completed status. Set restartPolicy: OnFailure or Never on Job Pods — Deployments use Always.

Parallel Jobs

Jobs can run multiple Pods in parallel for faster completion of large batch tasks:

spec:
  completions: 10    # Need 10 successful completions total
  parallelism: 3     # Run 3 pods at the same time

Kubernetes runs 3 Pods simultaneously. As each one completes successfully, it starts another until 10 total completions are reached. This speeds up a batch job that processes 10 chunks of data.

CronJobs: Scheduled Recurring Jobs

A CronJob creates a Job on a schedule — like a Linux cron tab, but managed by Kubernetes. Use CronJobs for nightly database backups, hourly report generation, daily cleanup scripts, or periodic health checks.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"    # Run at 2:00 AM every day
  concurrencyPolicy: Forbid # Do not start if previous run is still active
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: backup-tool:v1
            command: ["/bin/backup.sh"]

CronJob Schedule Format

┌──────────── minute (0–59)
│ ┌────────── hour (0–23)
│ │ ┌──────── day of month (1–31)
│ │ │ ┌────── month (1–12)
│ │ │ │ ┌──── day of week (0–6, 0=Sunday)
│ │ │ │ │
0 2 * * *   → 2:00 AM every day
*/15 * * * * → every 15 minutes
0 9 * * 1   → 9:00 AM every Monday

Inspecting Jobs and CronJobs

kubectl get jobs
kubectl get cronjobs
kubectl describe job database-migration
kubectl logs job/database-migration        # View output from job pods
kubectl get pods --selector=job-name=database-migration

To manually trigger a CronJob without waiting for the schedule:

kubectl create job --from=cronjob/nightly-backup manual-backup-test

DaemonSets vs. Deployments vs. Jobs

Key Points

• DaemonSets run exactly one Pod per node — ideal for logging agents, monitoring, and network plugins.
• Jobs run Pods to completion — when the task finishes successfully, the Job is done.
• Set restartPolicy: OnFailure on Job Pods, never Always.
• Use completions and parallelism to run batch tasks faster across multiple Pods.
• CronJobs use standard cron syntax to schedule recurring tasks at any frequency.