You cannot operate what you cannot see. Here is how to get useful signals out of Prometheus and turn them into alerts that mean something.
Prometheus scrapes metrics from HTTP endpoints, stores them as time series, and lets you query and alert on them with PromQL - the backbone of Kubernetes-era monitoring.
What to watch
- Keep label cardinality low and stable
- Alert on symptoms (latency, error ratio) via SLOs, not noisy causes
- Use recording rules for expensive dashboards and remote-write for long-term storage
- Track error rate and latency, not just up/down
- Alert on symptoms tied to user impact
Operate and inspect
Useful PromQL queries.
# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# error ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))Keep it healthy
- Put Prometheus behind auth/proxy; it has no built-in authentication
- Scope exporters and avoid exposing internal metrics publicly
Want senior help putting Prometheus to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.