Retrieval-augmented generation (RAG) grounds an LLM in your own documents by retrieving relevant chunks and putting them in the prompt. The model is the easy part - the hard, DevOps-shaped parts are ingestion, retrieval quality and evaluation.
The pipeline
# ingest: chunk -> embed -> upsert into a vector DB
chunks = split(document, size=800, overlap=100)
vectors = embed(chunks) # same model you will query with
vector_db.upsert(collection='docs', vectors=vectors, payload=metadata)
# query: embed question -> retrieve -> prompt
ctx = vector_db.search('docs', embed(question), top_k=4)
answer = llm(f'Use only this context:\n{ctx}\n\nQ: {question}')Where RAG goes wrong
- Different embedding models for index vs query - retrieval silently degrades
- Chunks too big (noise) or too small (lost context)
- No re-embedding plan when the model changes
- No evaluation, so a prompt tweak quietly breaks answers
Evaluate it like code, in CI
- run: python evals/rag_eval.py # scores answers vs a labelled golden set, fails under thresholdA RAG system is a data pipeline with a language model on the end. Treat it like a pipeline: version it, monitor it, and test it.
Put retrieval behind metrics and a golden-set eval in CI, and RAG becomes something you can operate and improve - not a demo that mysteriously regresses.