What makes or breaks your RAG pipeline

RAG breaks in stages

Like a chain, your RAG system is as strong as its weakest link. You may get away with it in development, but once you scale your system beyond test documents, these weaknesses become a real problem.

Large Language Models are great at content generation, but the quality of their output is dependent on the input. Give it wrong context, and it will generate the wrong output or, worse, make something up.

Standard RAG has 2 stages: retrieval and generation. If your retrieval is wrong, generation will fail - garbage in, garbage out.

Common RAG pitfalls

Most of these can be avoided by getting the fundamentals right, such as data cleaning and running evals.

Data Source

If you have the wrong data, everything collapses. No amount of clever chunking, reranking, or prompt engineering fixes a knowledge base that's stale, incomplete, or noisy.

A standard knowledge base is usually built by scraping data from the web, collecting documents, CRM records, or internal wikis. You can push this straight to the VectorDB, but that’s bound to fail at scale.

It’s important to clean and standardize the data in the ingestion stage. This will strip your indices of noise, reduce your context payload, and improve retrieval accuracy.

Chunking Strategy

Chunking is breaking down large docs into smaller "chunks" that can be retrieved individually and fit within a model's context window.

The goal is to split along natural semantic boundaries, so each chunk retains its semantic meaning. Split in the wrong place - mid-sentence, mid-paragraph, mid-table- and you get a chunk that reads like a fragment: technically retrievable, but has no semantic meaning.

LangChain provides several utilities for this. You can use text-based, length-based, or document-based strategies.

Embedding Models

Embedding models convert your text chunks into vectors.

A common mistake is treating the embedding model as a default, not a decision.

General-purpose embedding models underperform in domain-specific RAG pipelines.

Choose your embedding model deliberately, based on what your project actually needs.

Retrieval

When writing retrieval logic, regardless of the search metric, it’s common practice to limit the retrieved chunks to the top-k results.

This works well for most use cases but is vulnerable when working with large databases. Top-k assigns a relevance score solely based on the similarity score, and at scale this is not a reliable measure of relevance.

The solution is to use a reranker. A reranker is a model that considers each retrieved chunk and the query, then assigns a relevancy score.

Rerankers introduce context to your retrieval logic.

Closing remarks

We live in a world of agentic coding, and it’s easy to overlook the fundamentals. AI can scaffold your retriever, chunker, and embedding pipeline in minutes.

AI doesn’t ensure the knowledge base is clean or structured. It doesn’t decide the proper embedding model or ensure chunks retain their semantic meaning.

As a developer, you earn your keep by getting the fundamentals right.