When Prometheus misbehaves, the cause is usually one of a handful of familiar issues. Here is how to spot and fix them fast.
The usual suspects
- High-cardinality labels (user IDs, request IDs) exploding memory
- Treating Prometheus as long-term storage without Thanos/Cortex
- Alerting on raw values instead of rates/ratios tied to user impact
Commands for diagnosis
Useful PromQL queries.
# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))Prevent the repeat
- Keep label cardinality low and stable
- Alert on symptoms (latency, error ratio) via SLOs, not noisy causes
- Use recording rules for expensive dashboards and remote-write for long-term storage
Want senior help putting Prometheus to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.