Kubeflow and Argo Workflows are a common pairing in a modern DevOps stack. Here is how they fit together and a practical way to wire them up.
Where Kubeflow fits
Kubeflow runs the ML lifecycle on Kubernetes: Pipelines for orchestrating steps, training operators for distributed jobs, and KServe for scalable model serving - reusing your existing cluster and GitOps.
from kfp import dsl
@dsl.component
def train(lr: float) -> float:
return 0.94
@dsl.pipeline(name='train-pipeline')
def pipe(lr: float = 0.01):
train(lr=lr)Where Argo Workflows fits
Argo Workflows runs container-native, DAG or step-based workflows directly on Kubernetes - great for CI, data processing and ML pipelines where each step is a container.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: build- }
spec:
entrypoint: main
templates:
- name: main
dag:
tasks:
- { name: build, template: run, arguments: { parameters: [{ name: cmd, value: make build }] } }
- { name: test, template: run, dependencies: [build], arguments: { parameters: [{ name: cmd, value: make test }] } }
- name: run
inputs: { parameters: [{ name: cmd }] }
container: { image: alpine, command: [sh, -c], args: ['{{inputs.parameters.cmd}}'] }Making them work together
- Start with Pipelines + KServe rather than the whole platform
- Use GPU node pools with quotas and taints
- Set limits and activeDeadlineSeconds on templates
- Use workflow GC (TTL) to clean up finished runs
Want this pairing set up properly, with docs and handover? That is exactly the kind of work CodeWizard's embedded senior engineers do - and both Kubeflow and Argo Workflows have active community groups you can join for free.