This is the most commonly named platform on our list after the core frameworks — around 850 British employers mention it — and it is also the one where the job title conceals the widest range of actual jobs. “Databricks engineer” is used for data engineers, analytics engineers, ML engineers and platform administrators, and the four want different candidates.

It is also the only hire on this list where a significant part of the value is controlling what you spend, because unlike an open-source stack the meter is running.

Which of the four jobs are you advertising?

Decide before the advert goes out, because these people are not interchangeable and the market is competitive enough that a vague brief simply loses you time.

  • Pipeline building. Ingestion and transformation at scale — the bulk of the work in most organisations, and fundamentally a data engineering role.
  • Analytics enablement. Modelling data for consumption, governance, making the warehouse side usable by people who are not engineers.
  • Machine learning. Training and serving on the platform, using its lifecycle tooling.
  • Platform administration. Workspaces, access control, cluster policies and the spend. Frequently the unglamorous one that saves the most money.

If you need the first and hire the third, you get somebody bored by the work. If you need the fourth and hire the first, your bill grows quietly for a year.

Cost control is a technical skill here, and worth screening for

This is the part most interviews omit and most finance directors care about. The platform makes it very easy to spend money: a cluster left running overnight, an all-purpose cluster used for a scheduled job, everything on the most expensive runtime because nobody changed the default.

Ask directly what they have done to reduce a bill. Strong answers are specific — moving scheduled work onto job clusters that terminate, auto-termination policies, right-sizing after looking at actual utilisation rather than guessing, spot instances where the workload tolerates interruption, and cluster policies that stop people provisioning whatever they like. Somebody who has been handed an unexpected invoice will answer this with energy.

The related question is whether they can attribute cost to a team or a pipeline. Being able to say which workload is expensive is the precondition for fixing it, and in larger estates it is most of the job.

The Spark underneath, which is where real depth shows

The platform is comfortable enough that an engineer can be productive without understanding what is happening beneath, until something is slow — at which point the difference between a competent user and a strong one is stark.

Ask what they do when a job is slow. The answer you want moves through the query plan and the stage that is actually taking the time, toward data skew, partition counts, unnecessary shuffles, and whether a join could be broadcast. A candidate who reaches first for a bigger cluster has told you they will solve performance problems with your money.

Ask too about the storage layer: file sizes and the small-file problem, compaction, and what happens to a table that has been written to a thousand times a day for a year. These are the issues that arrive in year two and are invisible in a proof of concept.

Governance, which is increasingly the reason for the platform

Many organisations are on this stack because of governance rather than compute — one catalogue, lineage, and permissions that survive an audit. If that is your driver, screen for it rather than assuming it.

Useful questions: how they structured catalogues and schemas across environments, how access is granted and reviewed, and whether they have used lineage to answer a real question such as which reports break if this column changes. Somebody who has run a migration into a unified catalogue will have opinions about the ordering of that work, because doing it late is considerably more painful than doing it early.

What it pays, and why the market is tight

Indicatively £75,000 to £115,000 permanent for strong data engineering on the platform, higher for the machine learning and platform-administration variants, and day rates from around £550. It is one of the few technologies on this site where UK demand is genuinely deep.

That depth cuts both ways. Consultancies and the vendor’s own partner network absorb a great many of the experienced people, and certification is common enough that it no longer distinguishes anybody — treat it as evidence of familiarity rather than of capability. The differentiator in interview is the cost and performance material above, because it is the part certification does not cover and the part that shows whether somebody has run an estate rather than completed a course.

Ask how they feel about notebooks in production

This platform makes notebooks so convenient that a great deal of production logic ends up living in them, and that is where engineering standards quietly go. A notebook scheduled as a job is running code that is hard to test, awkward to review as a diff, and frequently full of cells whose order of execution matters in ways nobody has written down.

The answer to listen for is not that notebooks are forbidden — they are genuinely good for exploration, and a purist who refuses them will annoy your analysts. It is that production logic belongs in versioned modules with tests, imported by a thin notebook or job wrapper. Ask how they structured a repository, whether the transformations had unit tests, and what the review process was. Somebody who has cleaned up after an estate of scheduled notebooks will answer this at length and slightly wearily.

When the platform is not the point

Two redirections worth making honestly. If the actual need is training and serving models rather than moving data, the platform is incidental and the machine learning engineer guide describes the hire better. If the need is reproducibility and model governance specifically, the lifecycle tooling matters more than the compute and the MLflow engineer guide covers it.

Hiring for a platform rather than for a problem is the recurring error in this category. The platform is genuinely good; it is not the job.