Cassandra Monitoring
Monitoring a Cassandra cluster means continuously tracking the health of each node, the performance of read and write operations, and the state of background processes like compaction and repair. Good monitoring catches problems before they affect users and gives you the data to tune the cluster over time.
Monitoring Layers
Layer What to Watch ────────────────────────────────────────────────────────────── Cluster Node status, ring balance, gossip health Node CPU, heap memory, GC pauses, disk usage Operations Read/write latency, throughput, errors Background Compaction progress, repair status, streaming Application Query error rates, timeout rates, slow queries
Key Metrics to Monitor
Read and Write Latency
Metric Healthy Target ────────────────────────────────────────────────────────────── Read latency (p99) < 10 ms Write latency (p99) < 5 ms Coordinator read latency (p99) < 20 ms
p99 means the 99th percentile — 99% of operations finish within this time. A rising p99 while p50 stays flat indicates occasional slow outliers, often from GC pauses or compaction contention.
Pending Compactions and Repairs
Metric Warning Signal ────────────────────────────────────────────────────────────── Pending compaction tasks > 20 per node SSTable count per table > 50 (STCS) or > 10 (LCS) Repair last run > 7 days ago
Heap and GC
Metric Warning Signal ────────────────────────────────────────────────────────────── Heap used % > 75% sustained GC pause duration (p99) > 500 ms GC pause frequency > 5 pauses/min
Disk
Metric Warning Signal ────────────────────────────────────────────────────────────── Disk used % > 60% (leaves room for compaction) Commit log disk usage > 80% of commitlog partition
Monitoring with nodetool
# Overall node health: nodetool status nodetool info # Read/write latency histograms: nodetool tablestats ecommerce # Thread pool health: nodetool tpstats # Compaction progress: nodetool compactionstats # Streaming (during repair/bootstrap): nodetool netstats # GC stats: nodetool gcstats
JMX Metrics
Cassandra exposes all its metrics via JMX (Java Management Extensions). Monitoring tools connect to Cassandra's JMX port (default 7199) and poll metric values. Every metric visible in nodetool is available over JMX.
# Enable JMX remote access (for monitoring tools): # In cassandra-env.sh: JVM_OPTS="$JVM_OPTS -Dcom.sun.jmx.remote.authenticate=true" JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.0.0.1"
Prometheus and Grafana
The most common modern monitoring stack for Cassandra combines Prometheus (metrics collection) with Grafana (dashboards). The Cassandra Exporter or DataStax Metrics Collector bridges JMX metrics to Prometheus format.
Cassandra Node │ JMX port 7199 ▼ Cassandra Exporter (converts JMX → Prometheus format) │ HTTP /metrics endpoint ▼ Prometheus (scrapes and stores time-series metrics) │ PromQL queries ▼ Grafana (dashboards and alerting) │ ▼ Ops Team (alerts on Slack/PagerDuty)
Useful Prometheus Metrics for Cassandra
Metric Name (Prometheus format) Meaning ────────────────────────────────────────────────────────────── cassandra_client_request_latency_micros Read/write latency cassandra_compaction_pending_tasks Pending compaction backlog cassandra_storage_load Node data size cassandra_cache_hit_rate Key/row cache hit rate cassandra_jvm_memory_heap_used_bytes Heap consumption cassandra_jvm_gc_pauses_seconds_sum Total GC pause time cassandra_table_sstables_per_read_histogram SSTables read per query
System Log Monitoring
Cassandra writes important events, warnings, and errors to /var/log/cassandra/system.log. Monitor this log for tombstone warnings, GC pauses, dropped mutations, and schema disagreements.
# Watch the log in real time:
tail -f /var/log/cassandra/system.log
# Common warning patterns to alert on:
grep -E "WARN|ERROR|tombstone|GCInspector|DroppedMessages" \
/var/log/cassandra/system.log
Log Entries to Act On
Log Pattern Action Required ────────────────────────────────────────────────────────────── "tombstone cells... tombstone warn threshold" Redesign table or clear old data "GCInspector... paused ... ms" Tune JVM heap, reduce MemTable size "DroppedMessages ... MUTATION" Cluster overloaded; investigate writes "Schema version mismatch" Schema not yet propagated; wait/repair "Unable to gossip with any seeds" Seeds unreachable; check network "ERROR ... CommitLog" Disk full or permissions issue
Alerting Thresholds
Alert Threshold ────────────────────────────────────────────────────────────── Node down Any node DN in nodetool status Read latency p99 > 50 ms for 5+ minutes Write latency p99 > 20 ms for 5+ minutes Heap used > 80% for 10+ minutes Disk used > 70% Pending compaction tasks > 50 per node Last repair age > 10 days Dropped mutations rate > 0 per minute
Recommended Monitoring Tools
Tool Purpose ────────────────────────────────────────────────────────────── Prometheus + Grafana Metrics collection and dashboards AlertManager Alert routing and deduplication PagerDuty / OpsGenie On-call alerting ELK Stack Log aggregation and search DataDog All-in-one monitoring SaaS New Relic APM + infrastructure monitoring DataStax OpsCenter Cassandra-specific management UI
Summary
Monitoring Cassandra requires watching cluster node status, read/write latency, heap and GC behavior, disk usage, and background compaction. Use nodetool for quick checks and JMX-based tools like Prometheus with Grafana for continuous dashboards and alerts. Watch the system log for tombstone warnings, dropped messages, and schema mismatches. Set clear alerting thresholds and test that alerts fire before you need them in a real incident.