Search our crawl of 1.1 million UK careers pages for this exact title and you find nothing. No employer in that index advertises for an AI infrastructure engineer by name, though the work is being done across dozens of British companies under other labels.

So a keyword search will not find these people, and they may not recognise your advert either. Both problems dissolve once you are honest about the job, which is nearly always this: making a fixed quantity of compute produce more finished work per pound.

Nobody advertises this job under its own name

The titles that carry the work are platform engineer, ML platform engineer, infrastructure engineer, and site reliability engineer on a team that owns a training cluster. Search for the work, not the words.

The evidence is specific. Somebody who has run multi-node jobs through a scheduler, whether Slurm or Kubernetes with a queuing layer on top. Somebody who has debugged a distributed run that hung on a collective operation. Somebody who quotes throughput figures unprompted, because people who spent months moving a number remember it.

If your real problem is deploying and monitoring finished models, the MLOps engineer guide describes the person you want, and if you already run pipelines on Kubernetes the Kubeflow engineer guide is closer. This hire sits beneath both, where the compute bill is decided.

Three ways a cluster sits idle while you are paying for it

Nearly all wasted training spend comes from one of three places, and a candidate who cannot name them has not owned a cluster.

  • The accelerators are starved. The input pipeline cannot decode, augment and deliver batches as fast as the hardware consumes them, so expensive silicon waits on cheap CPU work and disk — the commonest fault, and the most often missed.
  • The job is communicating, not computing. At multi-node scale a growing share of step time goes on synchronising gradients across the interconnect, and whether that is tolerable depends on how the work is split and whether the communication overlaps the backward pass.
  • Memory is forcing a bad shape on the run. Activations will not fit, so batches shrink, and small batches let fixed per-step overheads dominate. The remedies — activation checkpointing, gradient accumulation, sharding optimiser state, mixed precision — all trade compute or complexity for headroom, and choosing between them is the skill.

None of that is kernel authorship, a scarcer discipline covered in the CUDA engineer guide — check your accelerators are fed before reaching for it.

Why the utilisation figure flatters a badly fed cluster

Utilisation is the number every dashboard offers, and it reports only that work was resident on the device rather than that the work was useful. A pipeline launching many tiny operations, or padding batches heavily, can show a reassuring ninety per cent while delivering half the throughput the hardware can manage.

Ask for four figures instead. Samples or tokens per second per accelerator, the only one that maps to finished work. The proportion of the hardware’s theoretical peak achieved, which shows the headroom left. The spread of step times rather than the mean, because the tail is where stalls hide. And the share of wall-clock time spent in successful steps once queue waits, failed runs and restarts are counted — routinely far worse than anyone expects, and the figure that reaches the invoice. A strong candidate volunteers three of those unasked.

Buying capacity, renting it, and the case for a reservation

This role usually owns the decision about where compute comes from. On-demand capacity is dear per hour and right when demand is genuinely bursty. Reserved capacity is much cheaper and pays only if you keep it busy. Owned hardware in a colocation facility is cheapest over a few years and brings every operational obligation with it, from spares to somebody whose weekend gets interrupted.

Duty cycle decides it. If your accelerators would be busy most of the month, reservation or ownership wins comfortably and the saving funds this hire several times over. If the pattern is a fortnight of intense work then six quiet weeks, a reservation only converts a variable cost into a fixed one. And duty cycle is not a given: it depends on whether anyone is queuing work, packing small jobs together and reclaiming idle capacity — exactly what this person does. Be wary of a standing answer: buy every time, or cloud every time, is a preference, not an analysis.

A ninety-minute conversation about one badly behaved job

Skip the systems design whiteboard. Ask them to talk you through a real job that ran worse than expected, and keep asking how they knew.

“How did you establish whether the input pipeline was the constraint?” You want an experiment rather than an opinion: feeding synthetic batches, timing the loader alone, watching whether step time moves when the data source changes.

“What happened to throughput per device when you doubled the node count?” Good answers are unembarrassed about sub-linear scaling and can attribute it.

“What is the longest a failed run has cost you, and what changed afterwards?” Checkpoint frequency, automatic restart, a pre-flight pass on a tiny slice.

Money, with the caveat that the adverts cannot help you

Because the title does not appear in adverts, no reliable benchmark exists and any precise band is extrapolation. Treat the figures below as indicative and calibrate against the platform salaries you already pay.

Permanent roles land broadly between £85,000 and £130,000, the upper end for people who have operated clusters of hundreds of accelerators rather than tens. Contract work runs roughly £550 to £900 a day, the premium going to candidates who can name a cost or throughput figure they personally moved. Expect to pay above your platform band: the same people are wanted by every team that has just bought hardware and found it half idle.

What the first fortnight should produce, and in what order

Ask for four things before anything structural changes: a baseline of throughput per accelerator on one real workload, a cost per completed run, a ranked list of where the time goes with evidence behind each entry, and one small fix shipped to show the numbers can be trusted.

If what comes back instead is a proposal to migrate the stack, you have hired an enthusiasm rather than an engineer. Insist on the ranked list: it turns an open-ended optimisation brief into a sequence of jobs with savings attached, and it tells you cheaply whether the problem deserves a permanent salary or was a configuration error all along.