From MLOps to AI engineering: what industrializing ML still teaches us

A trained model is the start of the work, and the discipline that gets it safely into production matters more than the model itself. In 2021 I wrote that MLOps was not a buzzword but a necessary discipline. Without a systematic approach, machine learning projects stall at proof of concept, or degrade quietly as live data drifts away from the data the model learned from.

Five years later, generative and agentic AI have widened the scope, and much of the industry is rediscovering those lessons under new names. I think the discipline now deserves a broader name, AI engineering, but the core ideas have changed less than the marketing suggests.

Why ML was never just software

The clearest statement of the problem is still Hidden Technical Debt in Machine Learning Systems, the 2015 NeurIPS paper by D. Sculley and colleagues at Google. Its famous diagram shows the model code as a small box surrounded by data collection, feature extraction, configuration, serving and monitoring. The paper names risks every platform team eventually meets: entanglement, hidden feedback loops, undeclared consumers and data dependencies. Its shorthand, Changing Anything Changes Everything, is the best one-line description of ML in production I know.

Three differences from ordinary software drove everything we built:

• More than code. Software lifecycles center on code. Machine learning also depends on data and models, which are harder to version and harder to diff.
• Experimental by nature. Development is a series of experiments that must be tracked to be reproducible.
• Never finished. After deployment, models need monitoring for drift and periodic retraining as well as bug fixes.

The lifecycle we industrialized

The lifecycle itself is familiar: data ingestion and preparation, feature engineering, training and tuning, evaluation against held-out data, deployment in the mode the use case needs (batch or real-time), then continuous monitoring with automated retraining when performance drops. In regulated industries you add lineage and audit trails across data, code and model artifacts.

At USAA we consolidated 360 data scientists onto one ML platform and cut model deployment from 8 weeks to 5 days. People assume the gain came from automation. Automation helped, but in my view the bigger shift was that model risk teams could validate models on the same platform data scientists built them on, using the same lineage and evidence. Validation stopped being a separate project that reassembled the facts afterwards. When evidence is a by-product of the pipeline, review gets faster and better at the same time.

What carries over to generative AI

Teams adopting foundation models sometimes conclude that MLOps no longer applies because they do not train the model. I disagree. You may not own the weights, but you own almost everything else: prompts, retrieval corpora, tool definitions, guardrails, evaluation sets and the routing logic between models. Each is a versioned artifact that changes behavior, and each needs the lineage we built for features and models.

The practical shift is that evaluation becomes the regression suite. In classical ML, a held-out dataset told you whether a retrained model was safe to promote. For generative systems the equivalent is a curated set of questions, expected behaviors and adversarial prompts, run automatically on every change to a prompt, a corpus or a model version. Teams that skip this are deploying on instinct. Teams that build it find that model upgrades, which used to be nerve-wracking, become routine.

Sculley's risks map almost one to one. Data dependencies become the document corpus behind retrieval, which drifts every time someone edits a policy page. Undeclared consumers become agents that discover and call your tools. Hidden feedback loops appear when AI-generated content flows back into the knowledge base the next answer draws from. And drift now includes a provider updating a model version underneath you, which can change behavior without a single line of your code changing.

What does not carry over

The analogy has limits, and teams that miss them get hurt. The first is determinism. A classical model given the same input returns the same score. A generative system can answer the same question differently on consecutive runs, so a single pass of your evaluation set proves less than it seems. Anthropic's guidance on agent evaluations recommends running each task several times and distinguishing between succeeding at least once and succeeding every time. For a customer-facing system in a bank, the second measure is the one that matters.

The second is grading. Held-out data came with labels. Many generative outputs have no single right answer, so teams use one model to grade another. That is workable, but the grader is now a model in production too, and it needs the same calibration against human experts that we demanded of any other model before we trusted its scores, plus an explicit way to say it does not know.

The third is cost. A classical model's inference cost was close to fixed. An agent that decides how many steps to take can make the same task many times more expensive on a bad day, so cost per task belongs on the monitoring dashboard next to accuracy and drift. Model risk teams that treat these three as footnotes will approve systems whose behavior they have only sampled once.

From MLOps to AI engineering

That is why I see AI engineering as one discipline that unifies DataOps, MLOps and DevOps pipelines across hybrid, multi-cloud and edge delivery, and across streaming and batch operations. Splitting them is what produces the gaps: a data team that changes a source without knowing which agent depends on it, or an application team that ships a prompt change without the evaluations an ML team would insist on.

The research points the same way. DORA's 2025 report found that AI amplifies whatever system it lands in, and its AI capabilities model lists healthy data ecosystems and AI-accessible internal data among the practices that make AI pay off. That is a DataOps finding as much as an AI one. When we moved hundreds of data scientists onto one governed platform, the hardest conversations were rarely about tools. They were about who owned a feature definition when three models depended on it.

Responsible design has to run through the whole lifecycle rather than being added at the end. The NIST AI Risk Management Framework, released in January 2023, organizes the work as govern, map, measure and manage, and NIST added a Generative AI Profile in July 2024. The value of a framework like this is less the vocabulary than the expectation that risk work happens continuously, alongside the build.

The cost of skipping this is becoming visible. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing cost, unclear value and inadequate risk controls. Those are the same reasons ML projects stalled at proof of concept a decade ago. The technology changed. The failure pattern did not.

The head start nobody talks about

Firms that built real MLOps platforms have an advantage they may not recognize. Lineage, evaluation, monitoring and shared evidence with risk partners are exactly what agentic systems need, and those capabilities take years to build and earn trust. Firms that skipped that work are about to pay for it twice, once for the models they already run and again for the agents they are about to deploy. If a provider changed the model under your most important AI use case tonight, how quickly would you know?