This is the scarcest hire on the site, and the reason is structural rather than fashionable. Writing CUDA well requires holding two models in your head at once: the mathematics of what you are computing, and the physical behaviour of a memory hierarchy. Most engineers are comfortable with one or the other. The intersection is small, knows it is small, and prices accordingly.
Before you start, it is worth being certain you need it, because the honest answer is often that you do not.
Three reasons to hire, and one bad one
The legitimate cases are reasonably clear. You have an operation with no good library implementation — something genuinely novel, or a fused sequence that would otherwise cross memory several times. You are working at a scale where a twenty per cent improvement pays a salary. Or your workload has an unusual shape that the standard libraries handle badly, which tends to mean irregular data, sparsity, or unusual precision requirements.
The bad reason is believing that hand-written kernels will beat well-chosen library calls in the general case. They usually will not. The vendor libraries are written by people who do this full time against hardware they have documentation for, and an engineer’s first honest act is often to tell you that the existing implementation is already near the roof.
Ask a candidate when they decided not to write a kernel. The good ones have that story, and telling it costs them nothing because their value is in the judgement rather than in the typing.
The questions that establish whether they think about memory
Performance here is almost always about data movement rather than arithmetic, and the interview should go straight there.
“Why might a kernel be slow even though the GPU reports high utilisation?” Utilisation is a famously misleading number. Strong answers reach for memory bandwidth saturation, uncoalesced access patterns, occupancy limited by register or shared-memory use, or warp divergence. Somebody who treats utilisation as the goal has read a dashboard rather than a profiler.
“Walk me through how you would use shared memory for a tiled matrix operation.” A standard problem, deliberately, because the point is to watch them reason about tile size, bank conflicts and synchronisation rather than to see whether they remember it. Fluency here is difficult to fake.
“What does your profiling workflow look like?” You want named tools and a habit of measuring before changing. An engineer who cannot describe how they find the bottleneck will optimise the part that was already fast.
The thing that separates good from excellent: knowing when to stop
Kernel optimisation has a long tail. There is almost always another fifteen per cent, and pursuing it can absorb months. The best engineers in this field have a clear sense of the roofline — what the hardware can theoretically deliver for this operation — and stop when they are close to it.
Ask how they knew a kernel was fast enough. The answer you want references a theoretical limit: achieved bandwidth against peak, or arithmetic throughput against the device’s capability. An engineer who says they kept optimising until the deadline is telling you they have no model of what good looks like, which makes their effort unbounded and difficult to plan around.
Where this role fits in the team, and the risk it creates
These engineers produce work that very few colleagues can review. That is a genuine organisational risk and it should be managed rather than ignored.
Insist on three things at the briefing stage. Correctness tests that compare kernel output against a reference implementation, because a subtly wrong kernel that is fast is worse than a slow correct one. Benchmarks committed alongside the code, so a later change that regresses performance is visible. And documentation of why the code is shaped as it is, since kernel code is dense and the reasoning is rarely apparent from reading it.
Without those, a departing contractor leaves behind something nobody can safely modify, and the practical outcome is that it gets rewritten in a library call a year later — which may have been the right answer all along.
What it pays, and why the number is high
Indicatively £100,000 to £160,000 permanent, with day rates from £800 and comfortably beyond for people with a track record in a recognised library or framework. Around 210 UK employers name it across the pages we index, but that figure includes a great many that merely want familiarity rather than authorship.
The premium is real and reflects genuine scarcity, not hype. The population that can do this work is largely employed by hardware vendors, quantitative finance, and the small number of organisations training large models — all of whom pay well and offer interesting problems. If your brief is a routine optimisation, expect to lose candidates to someone offering a harder one, and consider whether a contract engagement on a specific bottleneck serves you better than a permanent role you cannot keep interesting.
The adjacent hires that are usually the right answer
Most teams that think they need kernel authorship actually need somebody who can get more out of the hardware using existing tools. That is a different and much more available person, and the AI infrastructure engineer guide describes them.
If the specific problem is inference latency on NVIDIA hardware, the compiler route reaches most of the available gain without anybody writing a kernel, and the TensorRT engineer guide covers it. Try both before concluding you need this hire; if you still do, you will at least know precisely which operation to point them at, which is the difference between a six-week engagement and an open-ended one.