Most Prometheus pain is self-inflicted and predictable. Here are the mistakes we see most, and the fix for each.
Mistake 1: High-cardinality labels (user IDs, request IDs) exploding memory
Fix: Keep label cardinality low and stable. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.
Mistake 2: Treating Prometheus as long-term storage without Thanos/Cortex
Fix: Alert on symptoms (latency, error ratio) via SLOs, not noisy causes. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.
Mistake 3: Alerting on raw values instead of rates/ratios tied to user impact
Fix: Use recording rules for expensive dashboards and remote-write for long-term storage. Make it the default in your templates and enforce it in review or CI so the mistake cannot come back.
A safer baseline
Scrape config plus an alert rule.
# prometheus.yml
scrape_configs:
- job_name: app
static_configs: [{ targets: ['app:3000'] }]
---
# rules.yml
groups:
- name: app
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 10m
labels: { severity: page }
annotations: { summary: "5xx error rate above 5%" }Want senior help putting Prometheus to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.