Not one employer in our crawl of 1.1 million UK careers pages puts “RAG” in a job title. The word “retrieval” appears inside the job text often enough — buried in the fourth bullet of an AI engineer advert — but as an advertised role it does not exist. This is a practitioner’s term, used freely between engineers and almost never by whoever writes the requisition.

That gap has a consequence. The skill it names decides whether a retrieval system becomes a product or stays a demonstration, and because nobody advertises for it, hardly anybody screens for it either.

Good in a fortnight, stuck for six months

The pattern is consistent enough to plan around. A competent engineer wires up an embedding model, a vector store and a prompt inside ten working days, and the result is genuinely impressive on the questions somebody thought to try. Then it stops getting better. Half a year later the answers are roughly where they were in week three, the underlying model has been swapped twice, and somebody is drafting a paper arguing for fine-tuning.

The model is almost never the constraint. If the passage containing the answer never reaches the context window, nothing downstream can rescue it, and a stronger model simply produces a more fluent wrong answer — which is worse, because it is harder to spot. The plateau is a retrieval plateau almost every time, and most teams have no way of telling.

Did we find it, or did the model ignore it?

That question is the whole discipline in a sentence, and it is what to interview on hardest. A bad answer has two entirely different causes: the correct passage was never retrieved, or it was retrieved and not used. The remedies have nothing in common. The first is chunk boundaries, query handling and ranking. The second is prompt structure, the order of the context, occasionally a better model.

Ask a candidate how they told those apart on something they shipped. You want to hear that retrieved passages were logged for every query and inspected independently of the generated answer, so that for any bad output they could say whether the evidence was even present. Candidates who have only read about this discuss answer quality as one number, and one number cannot be debugged: it moves, and nobody can say which half moved it.

Making that measurement repeatable rather than impressionistic is a speciality in itself, which our model evaluation engineer guide covers.

The two interventions that usually break the deadlock

When a plateaued system does improve, it is normally because one of two things was added. Both are commonly claimed.

  • Reranking. Retrieve widely and cheaply, then rescore the top fifty or so with a model that reads query and passage together. It routinely does more for answer quality than anything done to the generation step, since only the first few passages are really read.
  • Hybrid search. Score lexically alongside vector similarity and fuse the two result sets. Embeddings handle exact tokens badly — part numbers, error codes, clause references, surnames, internal project names — and those are disproportionately what people search for.

Installed, or actually tuned?

Both amount to a few lines of configuration, so the CV tells you nothing. Numbers do. How many candidates went to the reranker, and how was that figure arrived at? What latency did it add, and what was given up to afford it? How were the lexical and vector scores combined, and at what weighting? Most usefully: which queries got worse?

Anybody who has genuinely tuned these has an answer to the last one, because every weighting change trades one class of query against another and somebody had to choose. Somebody who switched the feature on and watched the demonstration improve will not understand why you asked.

The corpus decides feasibility, so ask about it first

Four properties of your documents govern what any hire can achieve. Settle them before the brief is written, not in month two.

  • How large. Five thousand documents and five million are different engineering problems. Below a few thousand stable pages you may not need retrieval engineering at all, and an honest candidate will tell you so.
  • How often it changes. A static archive is indexed once. A corpus that churns hourly is a pipeline with freshness guarantees, deletions that must actually propagate, and stale answers as a permanent failure mode.
  • Who may read what. If the corpus mixes personnel files with public policy, permission filtering has to happen inside retrieval, per user, on every query. Bolting that on afterwards is close to a rebuild, and it is the question most often skipped.
  • Whether answers span documents. Questions that need relationships assembled across hundreds of sources are not a passage-retrieval problem, and the knowledge graph engineer guide describes that hire instead.

An hour with your own corpus, not a whiteboard

The exercise that sorts this pool is diagnostic rather than constructive. Give a shortlisted candidate a sanitised slice of your real documents, twenty real questions with the answers you expect, and a mediocre existing system. Ask them to explain, in an hour, why it is mediocre. Do not ask them to build anything.

What to watch: whether they look at what was retrieved before touching the prompt, whether they ask who asks these questions and why, whether they spot near-duplicates crowding the results. A strong candidate produces a short ordered list of suspects and says which to test first. A weak one rewrites the prompt, because that is the part they can see.

If the verdict is that the documents themselves are unusable — scanned, mis-parsed, stripped of their headings — the hire sits upstream of this one, and our LlamaIndex developer guide is the better start.

Indicative pay for a title nobody advertises

Indicatively £75,000 to £110,000 permanent, and a day rate usually between £550 and £750, more for somebody who has done this twice at scale. Treat both as soft, because there is no advertised market to benchmark against.

The absent title also distorts expectations. Candidates price themselves against whichever adjacent role they were last interviewed for — AI engineer, search or relevance engineer, machine learning engineer — so two people of comparable retrieval depth can be £25,000 apart on ask alone. Worth knowing that relevance engineers who ranked search results before language models arrived are often the strongest and cheapest applicants, because their CVs lack the fashionable vocabulary and they have spent years separating “we never found it” from “we found it and ranked it eleventh”.