Be clear with yourself about something before writing the advert: this is a Kubernetes hire. The machine learning vocabulary sits on top, but the person who succeeds in this role is somebody who is genuinely comfortable operating a cluster, and the ones who struggle are data scientists who learned the pipeline syntax.

That single reframing fixes most failed searches for this skill, because it changes where you look and what you test.

First, decide whether you need a platform at all

This is a heavyweight choice and it is frequently made for the wrong reason — because a larger company uses it, or because somebody wants the architecture on their CV.

The case for is real when several teams need to share accelerators fairly, when workloads must be scheduled and queued rather than run when somebody feels like it, when you need reproducible pipelines with lineage across many models, or when regulatory or data-residency constraints mean everything must run on infrastructure you control.

The case against is that you are taking on a distributed system with a significant operational surface, and that a managed service or a much simpler orchestrator will serve a team of five perfectly well. If one team trains three models a month, this will cost more in maintenance than it returns, and the engineer you hire will spend their time keeping the platform alive rather than improving anything.

Ask a candidate when they would advise against adopting it. Anybody who has operated one has a clear answer and usually gives it with feeling.

What the interview should actually test

Weight the Kubernetes far above the pipeline syntax, because the syntax is learnable in a week and the operational judgement is not.

“A training pod is stuck pending. Walk me through it.” This is the single most diagnostic question in the whole interview, because it is the most common real incident. You want resource requests against node capacity, GPU availability and device plugins, taints and tolerations, node selectors, quota, and a storage volume that cannot bind. Somebody who has run this answers fluently and has met at least three of those.

“How do you give six teams fair access to twelve GPUs?” Namespaces and quotas, priority classes and preemption, and an honest acknowledgement that fairness is a policy question the platform only enforces. Candidates who have not faced it describe a booking spreadsheet.

“What broke on your last upgrade?” The ecosystem moves and the components are numerous. An engineer who has done two or three upgrades will describe a version incompatibility between components, or a custom resource that changed shape. One who has never upgraded has been operating a fresh install.

The component sprawl is the hidden cost

What arrives is not one thing. There is pipeline orchestration, a notebook service, a serving layer, a metadata store, an experiment tracker, and usually an identity integration bolted to the side. Each has its own release cadence and its own failure modes.

Ask which components they actually ran, and which they turned off. This is revealing, because experienced operators run a deliberately small subset — frequently just pipelines and serving — and disable the rest to reduce surface area. A candidate who lists every component as though a full installation were the goal has probably inherited one rather than chosen it.

The related question is what they replaced. Plenty of mature teams keep the pipeline layer and swap the tracking component for something they prefer. That is a sign of judgement, not disloyalty.

Who does well in this role, and where to find them

The profile that works is a platform or infrastructure engineer who has become interested in machine learning workloads — somebody whose instincts are about reliability, resource management and blast radius, and who has learned what makes a training job different from a web service.

Those instincts transfer from anywhere Kubernetes is run seriously. A candidate who has operated a multi-tenant cluster for application teams will pick up the ML specifics quickly; the reverse conversion is slower and less reliable, because the operational knowledge takes years and the pipeline knowledge takes weeks.

If your actual need is the model lifecycle rather than the cluster underneath it, you may want a different hire entirely — the MLOps engineer guide covers that shape, and the AI infrastructure engineer guide covers the case where the real problem is getting more out of the accelerators you have.

Rates, and a realistic view of supply

Indicatively £85,000 to £125,000 permanent, day rates from about £600. Roughly 110 UK employers name it in the pages we index.

Supply is tighter than that number suggests, for a structural reason: the people who are good at this are good at Kubernetes generally, and Kubernetes generally pays well and has far more openings. You are competing not with other ML platform teams but with the entire platform engineering market, which is much larger and frequently offers a simpler life. That is worth knowing before you set a rate, and it argues for being specific in the advert about what is interesting here — accelerator scheduling and multi-tenancy are genuinely different problems from running stateless services, and the candidates you want are the ones who find that appealing.

A note on the exit route

If you adopt this, write down what it would take to move off it. Pipelines expressed in a platform-specific SDK are portable in principle and awkward in practice, and teams tend to discover the cost at the worst moment. An engineer who keeps the actual work in containers with thin orchestration around them has given you an exit; one who encodes business logic into platform-specific constructs has not. It is a reasonable thing to ask at interview, and the answer tells you how they think about lock-in generally.