RAG with LangChain: Building Retrieval-Augmented Generation

What is RAG?

RAG (Retrieval-Augmented Generation) is a technique for augmenting LLM knowledge with additional data. It involves bringing appropriate information from your data sources and inserting it into the model prompt, allowing the LLM to generate responses grounded in your specific documents, databases, or knowledge bases.

RAG has emerged as a popular mechanism to expand an LLM's knowledge base without fine-tuning. Documents retrieved from an external data source ground the LLM generation via in-context learning, enabling accurate answers about your proprietary data.

RAG Architecture: Two Main Components

Simple RAG in 40 Lines

The official LangChain tutorial demonstrates that you can create a simple indexing pipeline and RAG chain in approximately 40 lines of code. This makes RAG accessible even for beginners.

Basic Implementation Pattern

A typical basic implementation uses:

• An OpenAI LLM for generation
• A vector database like Weaviate or Pinecone for storage
• OpenAI embeddings for vectorization
• LangChain for orchestration

RAG Implementation Approaches

Approach 1: RAG Agent with Tools

The official LangChain tutorial recommends a RAG agent that executes searches with a simple tool as a good general-purpose implementation. The agent can:

• Decide when to retrieve information
• Refine queries based on initial results
• Perform multiple retrievals if needed
• Reason about which chunks are most relevant

This approach provides flexibility but requires more LLM calls, increasing latency and cost.

Approach 2: Two-Step RAG Chain

A two-step RAG chain uses just a single LLM call per query, which is fast and effective for simple queries. This approach:

• Retrieves relevant documents immediately
• Formats context with the query
• Generates answer in one LLM call
• Minimizes latency and cost

Best for straightforward question-answering over documents.

Advanced RAG Techniques

Parent Document Retriever

Advanced RAG uses techniques like Parent Document Retriever, which creates small and more accurate embeddings while retaining the contextual meaning of large documents. This solves a key RAG challenge:

• Small chunks: Better retrieval precision, but may lack context
• Large chunks: More context, but worse retrieval precision
• Parent Document Retriever: Retrieve with small chunks, return parent documents with full context

Hybrid Search

Combine semantic search (vector similarity) with keyword search (BM25) for best results. Many vector databases now support hybrid search natively.

Reranking

Retrieve more candidates than you need, then use a reranker model to select the most relevant ones. Dramatically improves quality with minimal latency increase.

Key Components in LangChain

Document Loaders

LangChain provides 100+ document loaders for PDFs, websites, databases, APIs, and more. Load data from virtually any source.

Text Splitters

Chunk documents intelligently. LangChain offers recursive character splitting, semantic splitting, and language-specific splitters that respect code structure.

Embeddings

Support for OpenAI, Cohere, HuggingFace, and open-source embedding models. Choose based on your quality, cost, and privacy requirements.

Vector Stores

Integrations with 50+ vector databases including Pinecone, Weaviate, Qdrant, ChromaDB, and more. Start with a local database for development, migrate to hosted for production.

Retrievers

Various retrieval strategies beyond simple similarity search: MMR (Maximum Marginal Relevance), similarity score threshold, and custom retrievers.

Production Best Practices

Chunk Size Optimization

Experiment with different chunk sizes. Common starting points are 500-1000 characters with 50-100 character overlap. Test with your specific data and queries.

Metadata Filtering

Store metadata (date, author, source, category) with chunks. Filter before semantic search to narrow the search space and improve relevance.

Query Transformation

Rewrite user queries before retrieval. Convert questions to statements, expand acronyms, or generate multiple query variations for better recall.

Cite Sources

Always return source documents with answers. Include page numbers, URLs, or document IDs so users can verify information.

Monitor and Iterate

Use LangSmith to trace RAG pipelines. Track which documents are retrieved, how often queries find no results, and user satisfaction. Continuously refine based on real usage.

Common Pitfalls