https://blog.landryzetam.net/tags/tsdb/Recent content in TSDB on Landry Zetam - Tech BlogHugo -- gohugo.ioen-usThu, 18 Dec 2025 00:00:00 +0000https://blog.landryzetam.net/posts/prometheus-wal-replay/Thu, 18 Dec 2025 00:00:00 +0000https://blog.landryzetam.net/posts/prometheus-wal-replay/<p>There&rsquo;s a special kind of frustration that comes from watching a pod restart 90 times over 37 hours, each time getting <em>just</em> a little further before Kubernetes kills it again. Today I debugged a Prometheus instance stuck in an infinite crash loop, and the root cause was surprisingly subtle: the database was simply too large to start within the default health check window.</p> <p>This is the story of startup probes, Write-Ahead Logs, and why your monitoring system might be the hardest thing to monitor.</p>