Retrieval that survives "tell me more"

Most RAG breaks the moment somebody types a follow-up, because a short follow-up has almost nothing to embed. The fix is two lines. The harder problem is that chunk size is not an independent choice, and nobody tells you that either.

By Jithin Roy

Full Stack Developer, Kerala, India

5 October 20268 min read

Almost every retrieval augmented generation tutorial does the same thing. Take a document, split it every five hundred tokens, embed the pieces, store them, and at query time embed the user's message and fetch the nearest few. It works in the demo because the demo asks one self-contained question.

Then a real visitor arrives and has a conversation. They ask about a project. They read the answer. They type: tell me more. And the whole thing falls over, because that phrase has essentially no semantic content. Embedded on its own it is a vector pointing at nothing in particular, and the nearest neighbours are whatever chunk happens to be closest to the origin of your particular corpus. The user gets a confident answer about the wrong thing.

The subject lives in the previous turn

Watch what actually happens in a conversation. Someone asks how a pricing engine works, gets an answer, and follows with how much did that cost. The word that is carrying the entire subject of the question, and it refers to something in the previous turn. Embedding this message alone throws away the only context that makes it answerable.

It is worth being clear that this is not a model problem. The generation model would handle the follow-up perfectly if it were given the right context. The failure is upstream, in retrieval, where the query being embedded is not the query the user is actually asking.

Expand the retrieval query, not the prompt

The fix is small enough to be easy to dismiss:

// Short follow-ups ("tell me more", "how much?") embed poorly on their own,
// so include the last couple of user turns in the retrieval query.
const retrievalQuery = [
  ...conversationHistory
    .filter(m => m.role === 'user')
    .slice(-2)
    .map(m => m.content),
  message,
].join('\n')

const embedding = await createEmbedding(retrievalQuery)

Three decisions in there are worth pulling out.

Only user turns are included. Assistant replies are long, fluent, and full of vocabulary the model chose rather than the user, so folding them in drags the query vector toward whatever the model happened to say last. The user's own words are a much better signal for what they want next.

Only the last two. Enough to carry a subject across a follow-up, few enough that a conversation which has moved on does not keep dragging the original topic along. If somebody asks about a pricing engine, then about a chat widget, you do not want the pricing engine still weighting the query five turns later.

And critically, this expansion applies only to the retrieval query. The generation call still receives the real message alongside the real conversation history. The model is never shown the concatenated blob; that string exists for exactly one purpose, which is to be turned into a vector.

Retrieval and generation want different inputs. Conflating them is the most common structural mistake in a RAG pipeline, and separating them is usually a two line change.

Chunk size is downstream of topK

Here is the part nobody says out loud. How big your chunks should be is not an independent decision you make by reading a blog post. It is determined by how many chunks you retrieve.

This system retrieves three:

const queryResponse = await index.query({
  vector: embedding,
  topK: 3,
  includeMetadata: true,
})

Three chunks is the entire evidence base for every answer. That constraint has a consequence that is obvious in hindsight and easy to get wrong in practice: one large chunk per subject will win every query about that subject and crowd out the specific answer.

Concretely. Suppose there is one two thousand word chunk describing a project end to end, and separately a short focused chunk about how that project prices things. A visitor asks a pricing question. The big chunk contains the pricing discussion too, along with everything else, so it scores well. It takes one of the three slots. Its neighbours are probably other parts of the same project. The specific pricing chunk, which is the one that would have produced the good answer, comes fourth and is never seen.

The knowledge base here is 49 narrow entries rather than a dozen broad ones, and that is not a stylistic preference. It is what topK 3 forces. If you retrieve twenty chunks and rerank, you can afford coarse chunking. If you retrieve three, you cannot.

Every chunk has to name its own subject

Related, and equally easy to get wrong when you write the chunks by hand.

A chunk that opens the valuation engine works by resolving age first reads perfectly to a human, because the human knows which project they are reading about. As a vector it is missing the strongest term in the query. When somebody asks how does SellPrime price a phone, the project name carries most of the semantic weight, and a chunk that never says it competes badly against one that does.

The rule that came out of it: write every chunk as though it will be read with no surrounding context, because that is exactly how it will be retrieved. Repeat the subject. It feels redundant when you are editing the file and it is what makes the chunk findable.

Upsert by id, and the orphan that outlives your edit

The embedding script writes by id, which makes reruns idempotent: change the text of an entry, run it again, and that vector is replaced.

What it does not do is delete. There is no reconciliation step, so the index is only ever added to. Rename an id and you have not moved anything, you have created a second vector; the old one is still there, still retrievable, and still holding text you have since decided was wrong.

That failure is particularly nasty in a RAG system because of what happens next. The orphan gets retrieved, gets concatenated into the context, and is handed to the model as fact. The model has no way to know one of its three sources is stale. It will state the outdated thing confidently, and the only trace is a vector nothing in your source tree references any more.

Keep ids stable, and after any rename or deletion, reconcile the live index against your source rather than assuming an upsert cleaned up after itself. An append-only store looks fine right up until it does not.

Make the embedding run resumable

Embedding a corpus is a loop over a paid API that can rate limit you, and a run that dies two thirds of the way through should not mean paying for the first two thirds again.

const PROGRESS_FILE = 'embedding-progress.json'

// ...per entry:
await index.upsert([{ id, values: embedding, metadata }])
await sleep(1000)                 // be civil to the API

// on a rate limit:
await sleep(60000)                // wait out the window, then continue
// on any other error:
await sleep(3000)                 // back off and keep going

A progress file plus upsert by id is all it takes. Because writes are idempotent, a resumed run that redoes a few entries is harmless, which means the progress file does not have to be perfectly accurate. That combination, idempotent writes and a rough bookmark, is much easier to get right than exactly-once processing and is almost always sufficient.

The differentiated backoff matters too. A rate limit is a temporary condition with a known shape, so waiting out the window and continuing is correct. Any other error is unknown, so a short backoff and a retry avoids turning one bad response into a stalled run.

How to tell whether any of this worked

Retrieval quality is easy to feel and hard to assert, and the temptation is to judge it by reading a few answers, which mostly measures how fluent the generation model is.

Look at the retrieval step directly instead. Write down a set of queries you care about, including the awkward ones, and for each record which chunks came back and with what scores. Do it before a change and after. That gives you a table you can actually reason about, and it separates two failure modes that look identical from the outside: the right chunk was never retrieved, versus the right chunk was retrieved and the model ignored it. Those have completely different fixes and no amount of prompt tuning will fix the first one.

Include follow-ups in that set. A query list made only of well-formed standalone questions will tell you everything is fine, which is precisely the blind spot this article started with.

What to take from this

Build the retrieval query from the recent user turns plus the current message. A bare follow-up has nothing to embed.
Use only user turns, and only the last couple. Assistant text and older topics both pull the vector off target.
Expand the retrieval query only. The generation call still gets the real message and the real history.
Choose chunk size from your topK, not from a rule of thumb. At topK 3 a broad chunk crowds out the specific one.
Name the subject inside every chunk. It will be retrieved with no surrounding context.
Upsert by id, keep ids stable, and reconcile after renames. An orphaned vector is handed to the model as fact.
Make embedding runs resumable with a progress file and idempotent writes.
Back off differently for rate limits and unknown errors.
Measure retrieval, not answers, and put follow-ups in the test set.

The general lesson is that retrieval quality is mostly a data shaping problem wearing a machine learning costume. The interesting decisions here were what to embed, how big to cut it, and what each piece has to say about itself. Nobody upgrades their way out of badly cut chunks, and no model is clever enough to use a document it was never shown. The projects that knowledge base describes are written up in the SellPrime and Synerggy Events case studies.