Prometheus WAL Replay: When Your Metrics Database Can't Start Fast Enough

6 min read
#Prometheus #Kubernetes #Observability #Troubleshooting #GitOps #TSDB
Observability
Table of Contents

There’s a special kind of frustration that comes from watching a pod restart 90 times over 37 hours, each time getting just a little further before Kubernetes kills it again. Today I debugged a Prometheus instance stuck in an infinite crash loop, and the root cause was surprisingly subtle: the database was simply too large to start within the default health check window.

This is the story of startup probes, Write-Ahead Logs, and why your monitoring system might be the hardest thing to monitor.

Part 1: The Symptoms

The Crash Loop From Hell

The alert was simple: Prometheus pod showing 1/2 containers ready with 90+ restarts:

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus
NAME                                            READY   STATUS    RESTARTS         AGE
prometheus-kube-prometheus-stack-prometheus-0   1/2     Running   90 (3m54s ago)   37h

The describe output told the story:

Warning  Unhealthy  3m22s (x5403 over 37h)  kubelet  Startup probe failed: HTTP probe failed with statuscode: 503
Normal   Killing    14m (x90 over 37h)      kubelet  Container prometheus failed startup probe, will be restarted

Following the Logs

The Prometheus container logs revealed what was happening during each restart:

ts=2025-12-18T12:40:48.804Z level=info component=tsdb msg="WAL segment loaded" segment=2715 maxSegment=4415
ts=2025-12-18T12:40:51.991Z level=info component=tsdb msg="WAL segment loaded" segment=2716 maxSegment=4415
ts=2025-12-18T12:40:55.258Z level=info component=tsdb msg="WAL segment loaded" segment=2717 maxSegment=4415

Prometheus was methodically loading WAL segments—4,415 of them—at roughly 3 seconds per segment. Quick math: that’s approximately 3.7 hours to complete the WAL replay.

The default startup probe? 15 minutes.

Part 2: Understanding the Problem

What is the WAL?

Prometheus uses a Write-Ahead Log (WAL) as part of its time-series database (TSDB). When metrics come in, they’re first written to the WAL before being compacted into blocks on disk. This provides durability—if Prometheus crashes, it can replay the WAL to recover recent data.

The WAL accumulates segments over time. Each segment is typically 128MB, and they’re only cleaned up after data is compacted into blocks. With 30 days of retention and continuous scraping from a busy cluster, you can easily accumulate thousands of segments.

Kubernetes Probes 101

Kubernetes has three types of probes:

ProbePurposeFailure Behavior
StartupWait for slow-starting containersKeep waiting until success or threshold exceeded
LivenessDetect deadlocked containersRestart the container
ReadinessControl traffic routingRemove from service endpoints

The startup probe is specifically designed for containers that need extra time to initialize. While it’s running, liveness and readiness probes are disabled. Once the startup probe succeeds, the other probes take over.

The default startup probe for Prometheus was:

startupProbe:
  httpGet:
    path: /-/ready
    port: http-web
  failureThreshold: 60
  periodSeconds: 15

This allows 60 × 15s = 900 seconds = 15 minutes for startup. Our WAL replay needed 3.7 hours.

The Vicious Cycle

Here’s what was happening every ~15 minutes:

  1. Pod starts, begins WAL replay
  2. Prometheus loads segments 0 → ~300 (15 minutes worth)
  3. Startup probe fails (Prometheus returns 503 because WAL replay isn’t done)
  4. Kubernetes kills the container
  5. New container starts, begins WAL replay from segment 0 again
  6. Repeat forever

The pod was making progress, but never enough to escape the probe timeout.

Part 3: The Fix

Finding the Right Configuration

My first instinct was to increase the startupProbe.failureThreshold directly in the Helm values:

# This doesn't work!
prometheus:
  prometheusSpec:
    startupProbe:
      failureThreshold: 1000

This had no effect. The Prometheus Operator doesn’t expose the startup probe directly through the Prometheus CRD spec.

After digging through the CRD documentation:

$ kubectl explain prometheus.spec.maximumStartupDurationSeconds

DESCRIPTION:
    Defines the maximum time that the `prometheus` container's startup probe
    will wait before being considered failed. The startup probe will return
    success after the WAL replay is complete.
    If set, the value should be greater than 60 (seconds). Otherwise it will be
    equal to 600 seconds (15 minutes).

The Prometheus Operator has a dedicated field for exactly this scenario: maximumStartupDurationSeconds.

Applying the Fix

The correct Helm values configuration:

prometheus:
  prometheusSpec:
    retention: 30d

    # WAL replay timeout - default is 600s (10min), insufficient for large TSDB
    # 4400 segments × ~3s each = ~13200s needed, set to 15000s (~4.2 hours)
    maximumStartupDurationSeconds: 15000

    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: nfs-csi-v2
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

After the Helm release reconciled, the startup probe transformed:

# Before
startupProbe:
  failureThreshold: 60
  periodSeconds: 15
  # Total: 15 minutes

# After
startupProbe:
  failureThreshold: 250
  periodSeconds: 60
  # Total: 4.17 hours

The Prometheus Operator automatically calculates appropriate failureThreshold and periodSeconds values from maximumStartupDurationSeconds.

GitOps Deployment

Following proper GitOps workflow:

# Edit the Helm release values
vim monitoring/controllers/base/kube-prometheus-stack/release.yaml

# Commit and push
git add monitoring/controllers/base/kube-prometheus-stack/release.yaml
git commit -m "fix: increase Prometheus WAL replay timeout"
git push

# Trigger reconciliation
flux reconcile source git flux-system
flux reconcile kustomization monitoring-controllers

Part 4: Prevention and Monitoring

Right-Sizing Your WAL

If your WAL is consistently large, consider:

  1. Reduce retention: Do you really need 30+ days in Prometheus? Consider using Thanos or Cortex for long-term storage.

  2. Increase scrape intervals: Scraping every 15s vs 30s doubles your data volume.

  3. Drop unnecessary metrics: Use metric_relabel_configs to drop high-cardinality metrics you don’t need.

  4. Compact more aggressively: Prometheus compacts WAL to blocks over time; ensure this is happening.

Monitoring Your Monitoring

Add alerts for Prometheus itself:

groups:
  - name: prometheus-health
    rules:
      - alert: PrometheusRestartingFrequently
        expr: increase(kube_pod_container_status_restarts_total{container="prometheus"}[1h]) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus is restarting frequently"

      - alert: PrometheusWALCorruption
        expr: prometheus_tsdb_wal_corruptions_total > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus WAL corruption detected"

Documenting Startup Times

Track your actual WAL replay time:

# Get timestamps from logs
kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | \
  grep "WAL segment loaded" | \
  head -1  # First segment

kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | \
  grep "Server is ready" | \
  head -1  # Startup complete

Document this in your runbooks so future-you knows what “normal” looks like.

Key Takeaways

  1. Startup probes exist for a reason — Containers with large datasets need time to initialize. Don’t assume defaults work for production workloads.

  2. The Prometheus Operator abstracts probe configuration — Use maximumStartupDurationSeconds instead of trying to configure the probe directly.

  3. Crash loops can be self-perpetuating — If startup never completes, the process never gets to clean up (compact WAL), making the next startup even longer.

  4. Monitor your monitoring — Prometheus can’t alert you about itself being down. Use external health checks or ensure your alerting can handle Prometheus restarts.

  5. WAL size correlates with startup time — High retention + high cardinality + frequent scrapes = long startup. Plan accordingly.

Appendix: Quick Reference

Check WAL segment count

kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  ls /prometheus/wal/ | wc -l

Estimate startup time

Segments × 3 seconds ≈ Startup time
4400 × 3s = 13,200s ≈ 3.7 hours

Prometheus CRD field

spec:
  maximumStartupDurationSeconds: 15000  # ~4.2 hours

Verify probe configuration

kubectl get statefulset -n monitoring prometheus-kube-prometheus-stack-prometheus \
  -o jsonpath='{.spec.template.spec.containers[0].startupProbe}' | jq .