Recommenders are probably the highest-revenue application of machine learning in existence, and the discipline is oddly unfashionable. It is also the one where the gap between an academic understanding and a working system is widest, because almost everything that makes a recommender succeed happens outside the model.
If you are hiring for this, the most useful thing to know is that candidates who talk mostly about algorithms are usually the weaker ones.
The feedback loop is the thing to interview about
A recommender changes the data it is trained on. It shows a user certain items, the user can only interact with what they were shown, and tomorrow’s training set is made of those interactions. Left alone, the system narrows: popular items get shown, get clicked, get more popular, and the long tail becomes invisible.
Ask a candidate how they handled that. Strong answers name it as a problem and describe deliberate counters — holding back a fraction of traffic for random or exploratory recommendations, logging what was shown and not only what was clicked, and correcting for position bias so a click at the top of a list is not treated as equivalent to one at the bottom.
Somebody who has not met this will describe improving the model against historical data, which is exactly the trap: offline metrics improve, the system gets narrower, and engagement quietly declines over months.
Offline metrics disagree with reality, and they should know it
This is the field’s defining frustration. A model that scores better on held-out historical data frequently performs worse in front of users, because the historical data records what the previous system chose to show.
Ask how they decided to ship something. The only reliable answer involves an online test with a business metric attached, and the good candidates will say so without embarrassment even though it makes their offline work sound provisional. They will also have a view on what to measure: click-through rate alone drives clickbait, so anything mature also watches longer-horizon behaviour — completion, repeat visits, revenue, returns.
The candidate who insists their offline improvement will translate has not shipped enough of these.
Cold start, which is where most systems are judged
New users and new items are the permanent condition rather than an edge case, and handling them is largely what separates a system that works commercially from one that works in a notebook.
Worth asking about both directions. For a new user: what do you show somebody with no history, and how quickly do you adapt once they act? For a new item: how does something with no interactions ever get surfaced, given that the model has no evidence for it? The answers involve content-based features standing in for behavioural ones, deliberate exploration budgets, and sometimes a simple rule that overrides the model for the first few impressions.
In marketplaces and publishing this is the whole problem, because the catalogue changes daily. Establish how fast yours turns over before you brief the role, because it changes who is suitable.
The architecture question, asked practically
Most production systems have two stages: retrieve a few hundred candidates cheaply from millions, then rank those precisely. Ask which stage they worked on, because the skills differ.
Retrieval is an engineering problem about approximate nearest neighbours, embeddings and latency budgets measured in single-digit milliseconds. Ranking is a modelling problem about features, and in practice it is frequently gradient-boosted trees rather than anything deep, because tabular features and speed both matter — which our XGBoost guide covers from the modelling side.
A candidate who has only worked on ranking will underestimate the retrieval constraint; one who has only worked on retrieval will underestimate how much of the quality comes from features. Either is fine if you know which half you are filling.
Where the real difficulty lives: features and latency
The unglamorous half of this job is serving features fast enough. A ranking model needs user features, item features and interaction features assembled in tens of milliseconds while somebody waits for a page.
Ask what their feature infrastructure looked like and where training and serving diverged. Training-serving skew is the classic defect here — a feature computed one way in a batch job and another way in the serving path, so the model sees something subtly different in production than it trained on. Anybody who has run a recommender has a story about this and can usually name the feature.
What it pays, and who is competing for them
Indicatively £80,000 to £130,000 permanent, with day rates from about £600, and materially higher at the consumer platforms where a percentage point of engagement is worth a great deal.
That is the competition to be aware of: the people who are genuinely good at this are concentrated in retail, marketplaces, streaming and social products, all of which can attribute revenue to the work directly and pay accordingly. If your recommender is a secondary feature rather than the product, expect to lose candidates to somebody for whom it is the product — and consider whether a contract engagement to build the first version is a better use of budget than a permanent role you cannot keep interesting.
A question worth asking before you hire at all
Ask yourself how you will measure whether the recommender worked. If you cannot run an online test — insufficient traffic, no experimentation platform, no agreed business metric — then you cannot evaluate this work, and a sophisticated recommender is a poor first investment.
In that situation a simple rule-based system, honestly described as such, often captures most of the available value and can be built in a fortnight. The engineers worth hiring will tell you that themselves, which is a useful thing to listen for.