← All insightsBest Practices

Prometheus best practices for teams

Nir Geier, CodeWizard · 26 May 2026 · 3 min read · Intermediate

Prometheus rewards a little discipline. These are the practices that keep it maintainable as a team and a codebase grow.

The practices

  • Keep label cardinality low and stable
  • Alert on symptoms (latency, error ratio) via SLOs, not noisy causes
  • Use recording rules for expensive dashboards and remote-write for long-term storage

What good looks like

Scrape config plus an alert rule.

yaml
# prometheus.yml
scrape_configs:
  - job_name: app
    static_configs: [{ targets: ['app:3000'] }]
---
# rules.yml
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 10m
        labels: { severity: page }
        annotations: { summary: "5xx error rate above 5%" }

Anti-patterns to retire

  • High-cardinality labels (user IDs, request IDs) exploding memory
  • Treating Prometheus as long-term storage without Thanos/Cortex
  • Alerting on raw values instead of rates/ratios tied to user impact

Want senior help putting Prometheus to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult