Entrada

The problem is that poor ML architecture rarely fails loudly. It fails expensively, and it does so in a way that’s almost impossible to attribute to a single decision. Across the engagements my team runs at Entrada, the most consistent finding isn’t that clients have bad models. It’s that they’ve built environments where good models slowly become liabilities – and nobody can point to the moment it happened.

This is what that looks like in practice, and how a properly designed Lakehouse-native architecture changes the equation.

The Four Costs You’re Already Paying

Most ML cost conversations focus on compute. Compute is the symptom, not the disease. The real costs of poor ML architecture sit in four places, and they compound.

1. The retraining tax

If you can’t reproduce a model from code – same data snapshot, same features, same parameters – every retraining cycle becomes a small archaeology project. Engineers dig through notebooks, hunt for “the version that worked,” and end up rebuilding pieces from memory.

Databricks recommends promoting code between environments rather than models, precisely so production training is reproducible by definition. When that discipline is missing, every retrain costs 3–5x what it should – and that’s before you count the trust erosion when results don’t match the previous run.

2. The drift tax

Models degrade. That’s not a flaw; it’s physics. Customer behavior shifts, upstream schemas change, a vendor updates an API. Without monitoring built into the architecture from day one, drift becomes something you discover from business stakeholders – usually after the model has been wrong for weeks.

The architectural fix isn’t a monitoring dashboard bolted on later. It’s treating inference logs, feature distributions, and model metrics as first-class data assets in the Lakehouse, queryable like any other table, with alerts wired to the same pipelines that govern your data quality.

3. The shadow infrastructure tax

This one is invisible until you go looking for it. Data scientists, blocked by slow handoffs, start building parallel stacks: a personal feature store, a private experiment tracker, a “temporary” pipeline that’s been running in production for two years. Each one is rational in isolation. Together they form a shadow architecture that nobody owns, nobody documents, and nobody can safely turn off.

I’ve walked into environments with eleven distinct ways to compute the same customer feature. Eleven. Each one slightly different, each one feeding a different model, each one a potential audit failure.

4. The governance tax

When models live outside a unified governance plane, every regulatory question becomes a manual investigation. Who trained this? On what data? With whose permission? When was it last validated? In a well-designed system these are SQL queries. In a poorly designed one, they’re week-long projects involving three teams.

Production-stage MLOps loop on Databricks – model serving, data profiling, and retraining all governed through the same Unity Catalog production catalog. This is the difference between governance as a property of the system and governance as a quarterly audit project.

This is exactly the gap Unity Catalog was built to close – and increasingly, it’s expanding beyond data into the agent and LLM layer too. Databricks recently extended Unity Catalog governance to AI agents via Unity AI Gateway, which means the same permissioning and audit model now covers LLM calls, MCP tool use, and cost attribution. If your architecture can’t plug into that, you’re going to be retrofitting governance instead of inheriting it.

What “Good” Actually Looks Like

The Databricks reference architecture for MLOps isn’t complicated – that’s the point. Good architecture is boring architecture: clear stages, clear promotion paths, clear ownership.

Three things in this diagram do more work than any tooling decision:

Environment separation. Dev, staging, and production are isolated catalogs in Unity Catalog with explicit access controls. Code moves between them through review. Models don’t get hand-carried from a notebook to prod.

Code-based promotion. The model that runs in production is trained by production code on production data. Not a copy of a dev model with the metadata changed. This single discipline eliminates an entire class of “it worked on my laptop” failures.

Unified governance. Data, features, models, and now agents all live under the same governance plane. Lineage isn’t something you generate for an audit; it’s a property of the system.

Where Teams Get Stuck

In our work at Entrada – much of it documented in our case study on optimizing legacy Databricks pipelines for a sales automation platform – we see the same three blockers over and over.

The first is feature reuse. Teams build features inside training notebooks, then re-implement them at inference time. The two implementations drift. Predictions degrade. Nobody can tell why because the drift isn’t in the model, it’s in the features feeding it. A governed feature store inside Unity Catalog solves this structurally, not procedurally.

The second is experiment hygiene. MLflow tracking exists in almost every Databricks environment we walk into. It’s used properly in maybe one in five. Without disciplined logging of parameters, metrics, artifacts, and data versions, you have a list of experiments but no way to compare them – which means you have a folder of runs, not a research program.

The third is the “we’ll productionize later” trap. A model gets built for a demo, generates buzz, and quietly becomes load-bearing. Six months later, the team that built it has moved on and nobody knows how to retrain it. This is the single most common origin story for the architecture audits we run.

The development stage in Databricks’ recommended MLOps workflow – every model artifact, parameter, and metric flows through MLflow tracking and registers to Unity Catalog, making “which version worked?” a query instead of an investigation.

A Practical Diagnostic

If you want to know whether your ML architecture is taxing you, three questions cut through most of the noise:

If your best data scientist quit tomorrow, could someone else retrain your top-3 production models from code, on a fresh cluster, with documented data sources – in under a day? If no, you have an architecture problem, not a people problem.
Can you answer “what data did this prediction depend on?” for any model in production, using a query rather than a meeting? If no, you don’t have lineage; you have hope.
When a model’s performance drops 5%, do you find out from your monitoring or from your customers? The answer determines whether you have observability or theater.

Teams that answer “yes” to all three have spent real effort on architecture. Teams that answer “no” to any are paying the tax described above, whether they see it in the budget or not.

This is the foundation underneath everything else we build – and it’s why we’ve invested in accelerators like the Governance Hub, which packages opinionated governance workflows directly into the Databricks workspace rather than asking clients to assemble them from parts. The pattern is the same as the MLOps story: good architecture isn’t a future state you migrate to. It’s the cheapest way to operate, starting now.

The Mindset Shift

The teams that get this right don’t treat ML architecture as a deliverable. They treat it as the operating system their AI practice runs on. They invest in it before the failure that would have forced them to. They measure success not by how many models they ship, but by how cheaply they can ship the next one.

That mindset – architecture as a discipline, not a phase – is the difference between an AI practice that compounds and one that just accumulates technical debt with neural networks attached.

The hidden costs of poor ML architecture aren’t actually hidden. They’re just denominated in time, trust, and opportunity cost rather than line items. Once you start looking, they’re everywhere.

The good news: so are the fixes.

Want a structured read on where your ML architecture sits today? Our AI + Data Maturity Assessment is built for exactly this conversation. For the canonical reference, Databricks’ Big Book of MLOps is the place I send every architect on my team.