Two hundred checks means two hundred scans. The checks are nothing — count nulls, count distincts, compare this period’s distribution to last period’s. The scan is the entire cost, and you’re paying it over and over to answer questions you could have answered together.

That’s not a figure of speech. On a managed platform you rent the cluster by the second, so the scan count is the invoice. Two hundred reads of the same table is two hundred times the compute for the same answers.

So batch them. Most checks against a table are aggregates over the same rows, so they fit in one pass: one scan, a pile of SUM(CASE WHEN ...) expressions, one result row with all the answers. Two hundred scans become one.

Then you find out what batching costs you.

Everything shares a fate now

Separate queries fail separately. Batched ones don’t, and that turns out to matter more than it sounds like it should.

Vectorized engines like Photon process data in columnar batches — that’s where the speed comes from. Some expressions can’t be vectorized. Python UDFs are the obvious one, since they force row-at-a-time serialization between the JVM and a Python worker. On a table with a few hundred million rows I’ve watched that turn a two-minute scan into an hour.

Put that UDF inside a batched aggregate and it doesn’t just slow down its own check. It de-vectorizes the whole query. Forty other checks were riding along in that scan and now they’re all on the slow path too.

The bill follows. A job that fell back doesn’t just finish late — it holds the cluster an order of magnitude longer to produce identical output. Same result, many times the money. Picture Bugs Bunny shoveling cash into a furnace, except the furnace is a compute cluster and the cash is your quarterly budget.

Two fixes

Compile the rules to SQL instead of calling out to code. Most of what people reach for a UDF to do — format validation, bucketing, some business rule — is expressible as CASE WHEN. The generated SQL is horrible to look at. It also stays in the vectorized engine, and that’s the difference between minutes and hours.

For the ones that genuinely need a UDF — decryption, mostly — pull them out of the batch and run them standalone. They’ll be slow. They were always going to be slow. But they’re slow by themselves instead of taking forty other checks down with them.

So the batch is a fast path with an entry requirement, and anything that can’t meet it gets its own query.

When it breaks

Worth knowing in advance, because the errors lie to you.

If something in a batched aggregate fails — bad cast, malformed expression — every check in the batch fails with the identical error. Forty-two failures across a dozen check types isn’t forty-two problems. It’s one broken expression and forty-one bystanders.

The check names are useless here. Go straight to the SQL fragment in the query trace; it usually names the expression and the value that broke it. Debugging by check name will take you somewhere unrelated.

The part that generalizes

The expensive thing usually isn’t the work. It’s the setup around the work — the scan, the round trip, whatever you had to load before you could start. Making the work faster gets you a few percent. Doing the setup once gets you the real number.

Which is also where the cost savings live, and why the big ones look implausible when you write them down. Nobody finds a 90% reduction by tuning a query. You find it by noticing the same table is being read a hundred times, or that a job has been quietly running unvectorized for months. The wins aren’t clever — they’re structural, and they were sitting there the whole time.

Sharing creates coupling, though. Once forty things depend on one scan, the question isn’t how fast that scan is anymore. It’s what can poison it.

Two hundred checks in one scan is easy. Keeping the two hundred and first from wrecking it is the actual job.

That’s not a dependency graph. It’s a stack of instructions that happens to usually work.

There’s no manifest saying what this thing needs. No resolution step that fails early if something’s missing. No version on any of it — just whatever happens to be sitting at that workspace path right now. Change one of those shared notebooks and everything that runs it changes too, silently, with nothing recording that it happened.

You can live with that for a while. What you can’t do is answer a question about it.

The question

Somebody asks what ran in production on the 14th.

With notebooks, here’s what you actually have: a job run history that says a notebook executed, and a workspace path. Whatever was at that path on the 14th is not necessarily what’s there now. If someone edited a cell, there’s no record. If they edited one of the notebooks it pulls in, there’s no record of that either, and no indication which version of it ran.

You cannot answer the question. Not “it’s hard to answer” — you genuinely do not have the information.

For a lot of teams that’s an annoyance. In a regulated environment it’s the whole ballgame. You need to know what code ran, who approved it, when it changed, and how to put it back. A notebook gives you none of those, and no amount of discipline within the notebook fixes it, because the gap is structural.

That’s why we moved. Not because notebooks are ugly or because DRY is nice — because we couldn’t produce an audit trail.

Wheels and bundles

The library became a versioned Python wheel. That part is straightforward: a wheel has a version number, a CI run that either succeeds or fails, and content you can diff. The stack of %run calls became ordinary Python imports that resolve up front or don’t resolve at all.

The more useful change was the Databricks Asset Bundle. A bundle declares everything about how the thing runs in one versioned file — the job, the compute it runs on, the trigger schedule, the library version it uses, the target environment. All of it lives in databricks.yml, in the repo, next to the code.

That’s the piece I’d underline. Before, the code was in one place and the job was in another. Cluster type, schedule, retry policy, permissions — those were console settings someone configured once and nobody could reconstruct. Now they’re in a file that goes through review like anything else. When someone changes the compute type, that’s a diff with a name on it.

The bundle also makes environments explicit. Dev, QAT, and prod are targets in the same file rather than three workspaces someone hopefully kept in sync.

The pipeline

The bundle describes what should exist. CI is what decides whether it gets to.

Our pipeline templates are pinned to a git tag — never a branch, so a pipeline that worked yesterday works today. Each environment gets validate, promote, and rollback jobs. Promotion to prod is gated manually on the default branch, so it’s a decision someone makes rather than a side effect of merging. State is Terraform-managed, which is what makes rollback a real operation instead of a hopeful one.

Auth runs through OAuth service principals rather than personal access tokens, which matters more than it sounds like it does: a PAT means every change is attributed to whoever generated the token, forever, including after they change teams.

And SonarQube gates coverage. A notebook has no meaningful notion of test coverage — there’s nothing to instrument and nothing to fail. Once the logic lives in a wheel, coverage is a number, that number has a floor, and dropping below the floor fails the pipeline. It stops being something people mean to get around to.

It also changes what the tests are for. When coverage is enforced rather than aspirational, the test suite ends up larger than the code it covers, and that ratio is the point — it’s what lets you change the thing without a knot in your stomach.

What it costs

It is slower. Editing a cell and hitting run gives you an answer in seconds; a merge request and a pipeline run does not. Anyone who tells you this migration is free is selling something.

The team’s objection was reasonable and I never pretended otherwise. What made it work was not taking notebooks away — the wheel exposes a Python API, so an analyst opens a notebook, imports it, and runs whatever they want interactively. Notebooks stayed as the interface. They stopped being the thing that runs on a schedule at 2am.

That distinction is what I’d have led with if I’d been sharper about it earlier. Most of the early resistance came from people hearing “no more notebooks,” which was never the proposal.

The other thing that helped: onboarding a new feed used to mean copying a notebook and hand-inserting a config row. Now it’s a menu-driven tool that generates the SQL and lets you review it before it applies. Nobody adopted this because of the audit story. They adopted it because their own work got easier and the audit story came along with it.

If you’re doing this

Start with the bundle, not the wheel. Getting the job definition into version control is most of the value and a fraction of the work — you can point a bundle at your existing notebooks on day one and still gain reviewable compute config, explicit environments, and real rollback. The packaging can come after.

Keep the interactive path working from the start. If people can’t import your library into a notebook and poke at it, you’re asking them to give up their feedback loop on faith.

And be honest that it’s slower. It buys you the ability to answer a question you currently cannot answer, which is worth it, but pretending there’s no tradeoff just makes people trust the rest of your argument less.

More on production data systems on the projects page.