PyTorch is the most-named AI framework in Britain. Across the 1.1 million UK careers pages we crawl, roughly 1,200 employers list it — more than any competitor — and almost none advertise it as a job title. It turns up in the requirements block, one bullet among eight, treated as a binary.

That binary is the problem. The range of ability behind the word is wider than for almost any other line on the page, because the framework is unusually pleasant to use badly. A competent Python developer can assemble a training loop from examples, watch the loss fall and quite fairly write PyTorch on their CV. Whether they can say what the machine was doing while it ran is the question that decides whether your training jobs finish.

Where the API ends and the framework begins

The API is small and the abstraction thin, which is much of why it won. Surface fluency arrives in a fortnight; the understanding underneath takes years; on paper the two look identical.

Three areas mark the boundary. The first is memory: what is held for the duration of a step and what is released, and why a model that trains happily at one batch size dies at the next one up. The second is the computation graph — that it is built as the forward pass executes, and that the tensors it retains are what it costs you, so detaching something, or failing to, changes what you can afford to run. The third is residency: whether a given tensor sits in host or accelerator memory, and what one careless line dragging it back across that boundary does to throughput. None of it is exotic; all of it is invisible in a tutorial.

Three probes into the layer below

Twenty minutes for all three, and the grade is in the reasoning rather than the vocabulary.

Why the backward pass is where memory runs out

Ask why a model can complete a forward pass and then fail on the backward one. A weak answer restates the question. A good one explains that the forward pass saved activations so gradients could be computed against them, that peak usage therefore arrives as the backward pass begins, and that gradients and optimiser state sit alongside the weights throughout.

Then ask what they would do, and listen for trades rather than a shopping list. Checkpointing buys memory with recomputation; accumulated batches buy it with time; reduced precision buys it with numerical headroom. The strongest read the allocator’s own memory summary first, because guessing which tensor is at fault is slower than looking.

What happens when the data loader cannot keep up

This is the commonest real cause of a slow training job, and the one candidates are least often asked about. Describe the symptom — accelerator utilisation bouncing around a third, step times that barely improve when the batch shrinks — and ask why. You want somebody who recognises a starved device and knows how to prove it: time the loop against tensors already in memory to find the ceiling, then compare. The rest is ordinary engineering — worker processes, CPU-side decoding, whether the job reads thousands of small files off network storage one at a time. The answer that should worry you is “ask for a bigger GPU”: a faster accelerator sitting idle is idle sooner.

When did you last read the source of something you use?

Ask it plainly. Anyone genuinely stuck has ended up inside the library’s own Python, and they will name the thing and say why. Candidates who have only worked at the level of the documentation have no answer, and that difference is most of what you are measuring.

Two identical runs that disagree

Run the same script twice with the same seed and the numbers can still differ. They should be able to explain why: floating-point addition is not associative, so a reduction performed in a different order gives a slightly different result, and several routines pick their order non-deterministically for speed. Add autotuning, worker seeding and a version bump, and small divergence is the default rather than a bug.

What matters more is what they do about it. Seeding the loader workers, pinning versions and switching on deterministic algorithms while debugging are table stakes. The real answer is that they establish the spread before believing anything: run the same configuration three times, see how far apart the results land, and refuse to celebrate an improvement smaller than that. That habit separates people who have run experiments that mattered from people who have merely run experiments.

Advertise the problem, not the library

Because this is a requirement and not a title, advertising for a “PyTorch Engineer” mostly attracts people who identify with a tool. Advertise the work instead: the domain, the scale of the data, how long a training run takes today and what you need it to take. Then state the framework requirement with its depth attached — that the person will be cutting the memory footprint of training jobs, or finding out why the accelerators are half idle. That filters harder than any keyword. The title you usually want is machine learning engineer, with the framework named underneath as a hard requirement.

The money follows the role, not the framework

Indicatively, engineers for whom this is a daily core requirement sit around £70,000 to £110,000 permanent in Britain, with contract day rates roughly £500 to £750, and research-adjacent London teams above both. Treat those as soft figures, because there is no market rate for a requirement. The surrounding role sets the price: the same skill is worth substantially more in a team training its own models than in one fine-tuning somebody else’s. Pay for the understanding described above, which is scarce, rather than for the word, which is not.

Give them a broken training script, not a blank page

The exercise that works is a short script that runs badly: it exhausts memory at a sensible batch size, the loader starves the device, and something is kept in the graph that should not be. Ask them to make it train faster inside the same memory budget, and to say what they expected each change to do.

An hour of that tells you whether they profile or guess, whether they change one thing at a time, and whether they can name what they would measure. It travels across frameworks, too: somebody who has worked at this depth in TensorFlow or JAX will pick the API up in a fortnight, because what took them years is exactly what the script tests.