Most LLM engineer hires that go wrong do not go wrong at the interview. They go wrong in the week before the advert, when nobody wrote down which of three quite different jobs the role actually is. The title has been stretched to cover an application developer building features on somebody else’s model, a model engineer who fine-tunes and serves weights, and a quality engineer whose entire output is the evaluation harness that tells you whether either of the others has made things better. All three legitimately call themselves LLM engineers. Almost nobody is strong at all three.

So this guide is organised around the decisions rather than the funnel. Which job you are hiring for, what separates production experience from prototype experience, how to test for it in under an hour, what it costs in Britain in 2026, and the failure modes that show up again and again once the person is in post.

Decide which of the three LLM engineer roles you are hiring

Write one sentence describing what the person will have shipped in six months. If you cannot, the rest of the process has nothing to aim at, and you will end up comparing candidates who are not comparable.

The application engineer

Builds product features on top of models they did not train: retrieval, tool calling, streaming interfaces, caching, fallback behaviour when the model is slow or wrong. Mostly a very good backend engineer who has learned to treat a model as an unreliable dependency. This is the role most companies actually need, and the one they most often over-specify by asking for a PhD.

The model engineer

Adapts and serves the weights: choosing between full fine-tuning and parameter-efficient methods, building the training data, quantising, and getting acceptable throughput out of a given amount of hardware. If you are not fine-tuning or self-hosting, you do not need this person yet, and hiring one to build a product feature wastes them.

The evaluation engineer

Owns the question “is it better than last week”. Builds the test sets, the scoring, the human review workflow and the regression gates. The least glamorous of the three and the one whose absence is most expensive, because without it every other change is a guess. On smaller teams this is a part of the application engineer’s job rather than a separate hire — but it has to be somebody’s job, explicitly.

If you genuinely need two of these, hire the application engineer first. They will tell you within a month whether the other one is really needed, and they will have the standing to be believed.

What separates production experience from prototype experience

Everybody’s CV now says they have shipped an LLM feature. The distinction that matters is whether they have shipped one, watched it fail in front of real users, and fixed it — because the second and third parts teach almost everything the first does not.

The cleanest single test is to ask how they knew a change was an improvement. Someone who has only prototyped will describe trying prompts and reading a few outputs. Someone who has run a system in production will describe a fixed test set, a scoring method, a threshold the change had to clear, and usually a story about a change that looked better and measured worse. That answer is hard to fake because it is a description of a habit, not of a fact.

Two further signals separate the groups quickly:

  • They talk about cost per request unprompted. Anyone who has owned a feature at volume knows roughly what a thousand requests cost and which lever moves it — caching, a smaller model for the easy cases, shorter context, batching. Candidates who have never held a budget find the question surprising.
  • They can name something they deliberately did not use a model for. Maturity in this field shows up as restraint: a lookup, a regular expression or a rules engine where a model would have been slower, dearer and less predictable. Candidates who want to put a model in everything have usually not yet had to support one.

Five things to screen for, and how to test each in under an hour

You do not need a take-home exercise that costs a strong candidate a weekend. Each of these is testable in conversation, and together they take about fifty minutes.

Retrieval, if retrieval is in scope

Ask what they would try first when a retrieval system returns plausible but wrong passages. Strong answers go to the retrieval step before the model: chunk boundaries, the embedding model’s fit for the domain, whether lexical search would have caught the term, and reranking. Weak answers reach for a bigger model or a longer prompt. If your documents are technical, contractual or multilingual, push harder here — general-purpose embeddings degrade on exactly that content, and knowing it is a real differentiator. Our guide to hiring a RAG engineer goes into this in more depth.

Failure behaviour

Ask what the feature does when the model is down, slow, or returns malformed output. You are listening for timeouts, retries with backoff, a degraded path, and idempotency on anything that writes. This is ordinary distributed-systems thinking and it is the most common gap in candidates who came to the field from data science rather than software engineering.

Evaluation

Ask them to design a test set for your actual product, out loud, in ten minutes. Watch whether they ask what a wrong answer costs you. A hallucinated citation in a legal summary and a clumsy sentence in a marketing draft are not the same problem, and the evaluation has to reflect that.

Security

Ask how they would stop a user making the system reveal another user’s data, or take an action on their behalf. Prompt injection has no clean fix, so the answer you want is architectural: least privilege on tools, no confidential content in a context a user can influence, confirmation on consequential actions, and output treated as untrusted. Candidates who claim a prompt can prevent it are telling you they have not thought about it.

Judgement about scope

Describe a feature you are considering and ask what they would cut. The best candidates will push back on the brief, narrow it, and say what they would measure first. This is the one signal that predicts whether they will be useful in the first month.

Interview questions that discriminate, and what good answers sound like

These are the four we would actually ask. Each has a failure mode that catches confident candidates with shallow experience.

“Walk me through a change you made that you later reversed.”

Strong: a specific change, the metric that looked good, the thing that got worse, and how they found out. Weak: no examples, or a reversal blamed entirely on someone else. Anybody who has run an LLM feature for six months has at least one.

“Your feature’s p99 latency doubled overnight and nothing was deployed. What do you look at?”

Strong: provider-side variance, context length creeping up with conversation history, cache hit rate falling, a retrieval index that has grown, queueing. Weak: straight to “switch models”. You are testing whether they instrument the system or guess at it.

“When would you fine-tune rather than improve the prompt or the retrieval?”

Strong: when you need a format or style reliably, when you have genuinely good labelled examples, when you are trying to shrink a model to cut cost — and an acknowledgement that fine-tuning rarely adds knowledge and can cost you general capability. Weak: fine-tuning as the default answer to quality problems, which usually means nobody has measured where the quality is actually being lost.

“How would you cut this feature’s cost by half without users noticing?”

Strong: measure first, then route easy requests to a smaller model, cache aggressively at the semantic level, trim context, batch where latency allows — and check quality after each step rather than at the end. Weak: a single lever, or no mention of measuring quality at all.

What it costs in the UK in 2026, and why the advertised market is so thin

Indicative permanent salaries in Britain, for someone who has genuinely shipped and supported an LLM feature: roughly £70,000–£95,000 at mid level, £95,000–£130,000 at senior, and above that in London for people with real training or serving experience at scale. Day rates for contract work typically run £500–£800, higher where self-hosting or performance work is involved.

Treat those as orientation rather than a benchmark, and here is the honest reason: almost nobody advertises this title. Across the live UK vacancies we observe, only a handful carry “LLM” in the job title at all — the work is overwhelmingly advertised as AI engineer, machine learning engineer or plain software engineer. Two consequences follow. Salary surveys for “LLM engineer” are built on samples too small to trust. And if you search only for that title you will miss most of the market, including people doing precisely this work under a different name.

The corollary is that your advert should probably not say “LLM engineer” either, or at least not only that. Name the work.

The failure modes we see most often

  • Evaluation theatre. A dashboard of scores nobody acts on, usually because the metrics were chosen for ease of measurement rather than for resembling the user’s complaint. The test is whether anyone has ever blocked a release on them.
  • Cost discovered in arrears. A feature ships, usage grows, and the bill arrives three months later. Ask for a cost-per-request estimate before launch, not after; a candidate who volunteers one is showing you how they work.
  • Retrieval that plateaus. Teams tune prompts for weeks when the retrieval step is returning the wrong passages. The symptom is a system that is confidently wrong rather than vague.
  • The rewrite at month four. A prototype built as a single prompt chain with no interfaces, no tests and no separation between orchestration and business logic. Often faster to rebuild than to extend, which is a hiring failure rather than an engineering one: the brief asked for a demo and got one.
  • Hiring a researcher for a product job. A strong publication record predicts almost nothing about shipping a reliable feature. If the role is genuinely research, the applied AI scientist and AI research engineer guides are the right ones.

Where these people actually are

Very few strong candidates are on a job board, because they do not need to be. In rough order of how well it works:

  • Adjacent internal talent. The backend engineer in your own company who has been quietly building with these tools for a year is often the best available hire, and needs a mandate rather than an offer.
  • Open-source contribution histories. Serving frameworks, evaluation libraries, retrieval tooling. A public commit history is the most honest CV in this field, and it is checkable before you ever speak to someone.
  • People doing the work under another title. Search on the work — retrieval, evaluation, inference, serving — rather than the title, for the reason described above.
  • Beyond your own market. The people who have trained and served these models at genuine scale are concentrated in a small number of places, and Britain is not one of them. That is the gap our practice exists to close, and the section below sets out how it works in practice.

A three-week process that does not lose the good ones

Strong candidates in this field are usually in two or three processes and will accept the one that moves. A month of silence between stages is how you lose them, and it is entirely self-inflicted.

  • Before you advertise: the one-sentence six-month outcome, which of the three roles it is, and who makes the decision.
  • Week one: a forty-minute technical conversation covering the five screens above. No take-home.
  • Week two: ninety minutes with the team, built around their work and a real problem from your backlog rather than a puzzle.
  • Week three: decision and offer. If you need a fourth stage, ask what it would tell you that the first three did not.

Two habits matter more than the structure. Give feedback within forty-eight hours, because in a market this thin it is remembered. And have whoever will actually work with them in the room by the second stage — candidates at this level are assessing you at least as hard as you are assessing them.