What ran in prod on the 14th?

We had a notebook whose entire job was to %run a stack of other notebooks before the real work started. Setup, config, helpers, shared functions — a dozen or so of them, in a specific order, and if you got the order wrong you found out somewhere around cell nine.

That’s not a dependency graph. It’s a stack of instructions that happens to usually work.

There’s no manifest saying what this thing needs. No resolution step that fails early if something’s missing. No version on any of it — just whatever happens to be sitting at that workspace path right now. Change one of those shared notebooks and everything that runs it changes too, silently, with nothing recording that it happened.

You can live with that for a while. What you can’t do is answer a question about it.

The question

Somebody asks what ran in production on the 14th.

With notebooks, here’s what you actually have: a job run history that says a notebook executed, and a workspace path. Whatever was at that path on the 14th is not necessarily what’s there now. If someone edited a cell, there’s no record. If they edited one of the notebooks it pulls in, there’s no record of that either, and no indication which version of it ran.

You cannot answer the question. Not “it’s hard to answer” — you genuinely do not have the information.

For a lot of teams that’s an annoyance. In a regulated environment it’s the whole ballgame. You need to know what code ran, who approved it, when it changed, and how to put it back. A notebook gives you none of those, and no amount of discipline within the notebook fixes it, because the gap is structural.

That’s why we moved. Not because notebooks are ugly or because DRY is nice — because we couldn’t produce an audit trail.

Wheels and bundles

The library became a versioned Python wheel. That part is straightforward: a wheel has a version number, a CI run that either succeeds or fails, and content you can diff. The stack of %run calls became ordinary Python imports that resolve up front or don’t resolve at all.

The more useful change was the Databricks Asset Bundle. A bundle declares everything about how the thing runs in one versioned file — the job, the compute it runs on, the trigger schedule, the library version it uses, the target environment. All of it lives in databricks.yml, in the repo, next to the code.

That’s the piece I’d underline. Before, the code was in one place and the job was in another. Cluster type, schedule, retry policy, permissions — those were console settings someone configured once and nobody could reconstruct. Now they’re in a file that goes through review like anything else. When someone changes the compute type, that’s a diff with a name on it.

The bundle also makes environments explicit. Dev, QAT, and prod are targets in the same file rather than three workspaces someone hopefully kept in sync.

The pipeline

The bundle describes what should exist. CI is what decides whether it gets to.

Our pipeline templates are pinned to a git tag — never a branch, so a pipeline that worked yesterday works today. Each environment gets validate, promote, and rollback jobs. Promotion to prod is gated manually on the default branch, so it’s a decision someone makes rather than a side effect of merging. State is Terraform-managed, which is what makes rollback a real operation instead of a hopeful one.

Auth runs through OAuth service principals rather than personal access tokens, which matters more than it sounds like it does: a PAT means every change is attributed to whoever generated the token, forever, including after they change teams.

And SonarQube gates coverage. A notebook has no meaningful notion of test coverage — there’s nothing to instrument and nothing to fail. Once the logic lives in a wheel, coverage is a number, that number has a floor, and dropping below the floor fails the pipeline. It stops being something people mean to get around to.

It also changes what the tests are for. When coverage is enforced rather than aspirational, the test suite ends up larger than the code it covers, and that ratio is the point — it’s what lets you change the thing without a knot in your stomach.

What it costs

It is slower. Editing a cell and hitting run gives you an answer in seconds; a merge request and a pipeline run does not. Anyone who tells you this migration is free is selling something.

The team’s objection was reasonable and I never pretended otherwise. What made it work was not taking notebooks away — the wheel exposes a Python API, so an analyst opens a notebook, imports it, and runs whatever they want interactively. Notebooks stayed as the interface. They stopped being the thing that runs on a schedule at 2am.

That distinction is what I’d have led with if I’d been sharper about it earlier. Most of the early resistance came from people hearing “no more notebooks,” which was never the proposal.

The other thing that helped: onboarding a new feed used to mean copying a notebook and hand-inserting a config row. Now it’s a menu-driven tool that generates the SQL and lets you review it before it applies. Nobody adopted this because of the audit story. They adopted it because their own work got easier and the audit story came along with it.

If you’re doing this

Start with the bundle, not the wheel. Getting the job definition into version control is most of the value and a fraction of the work — you can point a bundle at your existing notebooks on day one and still gain reviewable compute config, explicit environments, and real rollback. The packaging can come after.

Keep the interactive path working from the start. If people can’t import your library into a notebook and poke at it, you’re asking them to give up their feedback loop on faith.

And be honest that it’s slower. It buys you the ability to answer a question you currently cannot answer, which is worth it, but pretending there’s no tradeoff just makes people trust the rest of your argument less.