Two years ago this was not a role. It has become one because teams shipping language-model features discovered that nobody could answer the only question that mattered — is this version better than the last one — and that answering it properly is a full-time job requiring an unusual mixture of skills.

If you are hiring for it, you are further along than most. The reason to do so is simple: without this function, every other improvement is a guess, and a team can spend a quarter changing prompts with no idea whether anything got better.

What the job actually is

Not writing tests in the software sense, because the output is not deterministic and correctness is frequently a matter of degree. The work is building the apparatus that turns a judgement into a number you can act on:

  • A dataset of real inputs, held back from anything used to build the system, that resembles what users actually send rather than what the team imagined they would.
  • A scoring method per failure mode, which may be exact matching, a rule, a smaller model, another language model as a judge, or a human — usually several.
  • A threshold that blocks a release, agreed with whoever owns the product.
  • A human review workflow for the cases automation cannot settle, and enough consistency between reviewers that the results mean something.

The last two are where most attempts fail, and they are organisational as much as technical.

The dataset question comes first

Ask where their evaluation set came from. It is the most informative question because almost everybody gets it wrong the first time.

The weak answer is that the team wrote some test cases. Those reflect what engineers imagined, cluster around the happy path, and miss the phrasings that actually break things. The strong answer is that it was sampled from production traffic, stratified to include the difficult and rare categories deliberately rather than proportionally, and grown whenever a new failure was found in the wild.

Follow up by asking how they kept it honest. An evaluation set that the team optimises against for six months stops measuring quality and starts measuring familiarity, which is the same overfitting problem as in ordinary modelling and is much less often noticed here.

Using a model as a judge, and its limits

Scoring open-ended output with another language model is now standard practice, and a candidate should be able to discuss both why it works and where it misleads.

The known failure modes are worth hearing named: judges prefer longer answers, prefer their own style of writing, are sensitive to the order options are presented in, and are unreliable on anything requiring domain expertise they do not have. The counters are equally concrete — calibrating the judge against human labels on a sample, randomising presentation order, scoring specific attributes rather than asking for an overall verdict, and never using a judge from the same family as the model being judged where that can be avoided.

Somebody who treats an automated score as ground truth has not validated their instrument, and that is the whole job.

What separates evaluation from evaluation theatre

The distinction is whether anything has ever been stopped. Ask directly: has a release been blocked by these numbers, and what happened next?

A candidate who can describe holding back a change that the team wanted to ship, and being right, is describing a function with real authority. One whose dashboards were admired and never acted upon has built a decoration. The follow-up worth asking is who owned the threshold, because an evaluation function that cannot say no is advisory, and advisory quality functions do not survive a deadline.

The second tell is cost. Running a comprehensive evaluation on every change is expensive, so mature practice has tiers: a fast cheap set on every commit, the full set before release, and human review on a sample. Somebody who has run this at scale will volunteer that structure.

Who is good at this, and why they are hard to find

The profile is genuinely unusual. It needs enough statistical literacy to reason about sample sizes and significance, enough engineering to build the harness and put it in a pipeline, enough product sense to know which failures matter commercially, and enough diplomacy to tell a team their release is not ready.

In practice the people who do it well often come from testing and quality engineering backgrounds rather than from machine learning — they already think in terms of coverage, reproducibility and defect classes, and the statistical part is more learnable than the temperament. Somebody who has run a human annotation operation for any purpose is also a strong candidate, because inter-rater agreement is the skill that most reliably transfers.

If your need is broader than evaluation — the whole lifecycle including deployment and monitoring — the MLOps engineer guide is the closer fit, and on a smaller team this is frequently part of that role rather than a separate hire. It should still be somebody’s explicit responsibility, named as such.

Pay, and the one market on this site where knowing the problem is an advantage

Indicatively £75,000 to £115,000 permanent, with day rates from around £550.

The title barely appears in British job adverts, which means two things. There is no benchmark, so any figure including this one is inferred from adjacent roles. And there is very little competition for the people who can do it, because most employers have not yet worked out that they need the function — which makes this one of the few areas on this site where an employer who understands the problem has an advantage in hiring rather than a disadvantage.

Advertise it as building the thing that decides whether the AI works. The people who want that job recognise the description immediately, and they are usually already frustrated somewhere that does not have one.