← All insightsBest Practices

Common Prometheus mistakes and how to avoid them

CodeWizard Academy · 21 May 2026 · 3 min read · Intermediate

Most Prometheus pain is self-inflicted and predictable. Here are the mistakes we see most, and the fix for each.

Mistake 1: High-cardinality labels (user IDs, request IDs) exploding memory

Fix: Keep label cardinality low and stable. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.

Mistake 2: Treating Prometheus as long-term storage without Thanos/Cortex

Fix: Alert on symptoms (latency, error ratio) via SLOs, not noisy causes. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.

Mistake 3: Alerting on raw values instead of rates/ratios tied to user impact

Fix: Use recording rules for expensive dashboards and remote-write for long-term storage. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.

A safer baseline

Scrape config plus an alert rule.

yaml
# prometheus.yml
scrape_configs:
  - job_name: app
    static_configs: [{ targets: ['app:3000'] }]
---
# rules.yml
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 10m
        labels: { severity: page }
        annotations: { summary: "5xx error rate above 5%" }

Want senior help putting Prometheus to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult