← All insightsAI

RAG in production: a DevOps view of retrieval-augmented generation

Nir Geier, CodeWizard · 30 June 2025 · 3 min read · Intermediate

Retrieval-augmented generation (RAG) grounds an LLM in your own documents by retrieving relevant chunks and putting them in the prompt. The model is the easy part - the hard, DevOps-shaped parts are ingestion, retrieval quality and evaluation.

The pipeline

python
# ingest: chunk -> embed -> upsert into a vector DB
chunks = split(document, size=800, overlap=100)
vectors = embed(chunks)                 # same model you will query with
vector_db.upsert(collection='docs', vectors=vectors, payload=metadata)

# query: embed question -> retrieve -> prompt
ctx = vector_db.search('docs', embed(question), top_k=4)
answer = llm(f'Use only this context:\n{ctx}\n\nQ: {question}')

Where RAG goes wrong

  • Different embedding models for index vs query - retrieval silently degrades
  • Chunks too big (noise) or too small (lost context)
  • No re-embedding plan when the model changes
  • No evaluation, so a prompt tweak quietly breaks answers

Evaluate it like code, in CI

yaml
- run: python evals/rag_eval.py   # scores answers vs a labelled golden set, fails under threshold
A RAG system is a data pipeline with a language model on the end. Treat it like a pipeline: version it, monitor it, and test it.

Put retrieval behind metrics and a golden-set eval in CI, and RAG becomes something you can operate and improve - not a demo that mysteriously regresses.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult