Skip to content
Muthu Bommiah

Initiative · USAA · 2017 – 2022

Enterprise Machine Learning Platform

An enterprise MLOps platform that moved 360 data scientists onto one governed environment, saved $25M in license costs over 3 years and cut model deployment from 8 weeks to 5 days.

360data scientists on one governed platform
$25Mlicense savings over 3 years
8 wks → 5 daysmodel deployment time

Problem

Data scientists worked in non-compliant sandboxes and on desktops across many tools. Models took weeks to deploy, and Risk had no consistent way to validate them.

Approach

  • Architected a unified platform (Domino, Airflow, cloud-native) managing the full model lifecycle across Python, R, SAS and H2O.
  • Built deployment pipelines for on-demand, batch, event-driven and automated retraining workloads; introduced Apache Airflow at USAA.
  • Created a one-click Model Deploy Console with audit traceability.
A loop from data to features, training, a highlighted validate and risk review step, deployment and monitoring, with monitoring feeding back to data for retraining.DataFeaturesTrainValidate &risk reviewDeployMonitorretrainone governed path
The governed path from idea to production. Validation and risk review sit inside the loop, so every retrain passes the same gate.

Governance

  • The platform became Model Risk Management's shared basis for independent model validation.
  • Helped create the first-line model risk operating model with the Enterprise Model Risk Management Committee.
  • Per-model drift monitoring feeding automated retraining.

Outcome

  • 360 data scientists and modeling analysts consolidated; $25M saved in license costs over 3 years.
  • Model deployment time cut from 8 weeks to 5 days; 22 models to production in 6 months across 3 lines of business.
  • Real-time scoring for 5 cyber financial-crime models at 24M+ scores per day, under 500 ms.

Lesson

A trained model is only the beginning. Value comes from industrializing deployment, monitoring and validation.