What Is RAG (Retrieval-Augmented Generation)

Here's the problem RAG solves: language models know a lot about the world but nothing about your business. They can write great marketing copy but can't tell a customer your refund policy. They can analyze a dataset but don't know your company's revenue targets. RAG fills this gap by connecting the model to your private knowledge base.

I use RAG in virtually every agent I build. Support agents with RAG resolve 80%+ of tickets because they have access to the full knowledge base. Sales agents with RAG quote accurate pricing and feature details because they retrieve current product information. Internal operations agents with RAG follow company SOPs because the procedures are in their retrieval pipeline.

The setup isn't complicated. You process your documents into chunks, store embeddings in a vector database, and add a retrieval step before the model generates its response. Supabase with pgvector handles this for most of my client deployments without needing an additional service. The investment is a day or two of setup that pays off every single time the agent gives a correct, grounded answer instead of hallucinating.

The RAG pipeline has three steps that happen in sequence on every query. First, the system takes the user's question and converts it into a numerical representation (embedding) that captures its meaning. Second, it searches a vector database for stored documents or data chunks whose embeddings are closest in meaning to the query. Third, it feeds the retrieved documents into the language model's context along with the original question, and the model generates a response grounded in that specific information.

This might sound abstract, so here's a concrete example. A customer asks your support agent 'What's your refund policy for annual plans?' Without RAG, the model either makes up an answer based on general training data or says it doesn't know. With RAG, the system retrieves your actual refund policy document from the vector database, includes it in the model's context, and the agent responds with your exact policy. The difference between a hallucinated answer and a correct one.

Setting up RAG involves three components: a document processing pipeline, a vector database, and a retrieval layer. The document pipeline takes your source material (PDFs, web pages, documentation, database records), splits it into chunks, generates embeddings for each chunk, and stores them in the vector database.

The vector database (Supabase with pgvector, Pinecone, Weaviate, or Chroma) stores these embeddings and enables fast similarity search. When a query arrives, the retrieval layer converts the query to an embedding, searches the vector database for the most similar chunks, and returns the top results to include in the model's context.

Chunking strategy matters more than most people realize. Too large and the retrieved content wastes context tokens on irrelevant information. Too small and the chunks lack the context needed for a good answer. I typically use 300-500 token chunks with 50-token overlap between adjacent chunks.