← All insightsObservability

Monitoring and observability for Kubeflow

Nir Geier, CodeWizard · 10 September 2025 · 3 min read · Intermediate

You cannot operate what you cannot see. Here is how to get useful signals out of Kubeflow and turn them into alerts that mean something.

Kubeflow runs the ML lifecycle on Kubernetes: Pipelines for orchestrating steps, training operators for distributed jobs, and KServe for scalable model serving - reusing your existing cluster and GitOps.

What to watch

  • Start with Pipelines + KServe rather than the whole platform
  • Use GPU node pools with quotas and taints
  • Enable step caching and version pipeline artifacts
  • Track error rate and latency, not just up/down
  • Alert on symptoms tied to user impact

Operate and inspect

bash
kubectl -n kubeflow port-forward svc/ml-pipeline-ui 8080:80
# compile and submit
python -c "from kfp import compiler,dsl; compiler.Compiler().compile(pipe,'pipe.yaml')"

Keep it healthy

  • Isolate teams with namespaces and RBAC
  • Secure the dashboard behind SSO

Want senior help putting Kubeflow to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult