Skip to main content

​
What is RAG?

Retrieval-Augmented Generation (RAG) combines a vector search step with an LLM generation step:
  1. Retrieve — find document chunks most relevant to the user’s question
  2. Augment — inject those chunks into the LLM prompt as context
  3. Generate — the LLM answers based on the retrieved context, not just its training data
This lets your chatbot answer questions about your specific documents — product manuals, FAQs, legal documents, internal wikis — without retraining any model.

​
How LangChat’s pipeline works

​
1. Standalone question reformulation

Before searching, LangChat uses the LLM to rewrite the user’s message as a standalone query. This resolves pronouns and references to earlier messages:
The reformulated question is then embedded and searched.

​
2. Embedding

The question is embedded using OpenAI’s text-embedding-3-large model (3072 dimensions). The same model must be used when indexing documents — mixing models produces incorrect results. LangChat queries Pinecone for the top-k most similar chunks (k=5 by default via the retriever). The similarity metric is cosine distance.

​
4. Flashrank reranking

The top-k Pinecone results are reranked by Flashrank, a fast cross-encoder model that more accurately scores relevance than cosine similarity alone. The default model is ms-marco-MiniLM-L-12-v2, keeping the top 3 results. Reranking improves answer quality significantly — especially for long documents where many chunks may be superficially similar but only a few are truly relevant.

​
Pinecone namespaces

Use namespaces to partition documents within a single index. Searches are scoped to the namespace you configure:
Namespaces are useful for:
  • Separating different clients in a multi-tenant app
  • Partitioning by language or region
  • Separating document types (e.g., products vs. policies)

​
Changing the retriever depth

The default retriever fetches k=5 chunks before reranking. To fetch more candidates before reranking (improves recall at the cost of latency): This is controlled by the PineconeVectorAdapter internals. For advanced customization, see Extending Adapters.

​
Embedding model choice

Configure the embedding model on the Pinecone provider:
You must use the same embedding model for both indexing and retrieval. If you change the model, re-index all documents with the new model in a fresh Pinecone index.

​
When there’s no relevant context

If Pinecone returns no relevant results (low similarity scores), the LLM still receives the prompt — but the {context} placeholder will be empty or contain low-quality chunks. This can lead to hallucinated answers. Best practices:
  • Always tell the model what to do when context is missing: “If the answer is not in the context, say you don’t know.”
  • Ensure documents are indexed before going live
  • Monitor queries that return empty context (visible in Supabase request_metrics)