# Enterprise Machine Learning Platform

USAA · 2017 – 2022 · Role: Lead Software Engineer, then Staff Engineer

> An enterprise MLOps platform that moved 360 data scientists onto one governed environment, saved $25M in license costs over 3 years and cut model deployment from 8 weeks to 5 days.

- **360** data scientists on one governed platform
- **$25M** license savings over 3 years
- **8 wks → 5 days** model deployment time

## Problem

Data scientists worked in non-compliant sandboxes and on desktops across many tools. Models took weeks to deploy, and Risk had no consistent way to validate them.

## Approach

- Architected a unified platform (Domino, Airflow, cloud-native) managing the full model lifecycle across Python, R, SAS and H2O.
- Built deployment pipelines for on-demand, batch, event-driven and automated retraining workloads; introduced Apache Airflow at USAA.
- Created a one-click Model Deploy Console with audit traceability.

## Governance

- The platform became Model Risk Management's shared basis for independent model validation.
- Helped create the first-line model risk operating model with the Enterprise Model Risk Management Committee.
- Per-model drift monitoring feeding automated retraining.

## Outcome

- 360 data scientists and modeling analysts consolidated; $25M saved in license costs over 3 years.
- Model deployment time cut from 8 weeks to 5 days; 22 models to production in 6 months across 3 lines of business.
- Real-time scoring for 5 cyber financial-crime models at 24M+ scores per day, under 500 ms.

## Lesson

A trained model is only the beginning. Value comes from industrializing deployment, monitoring and validation.

Technologies: Domino, Apache Airflow, Kafka, OpenShift, Python, R, SAS, H2O, Prometheus, Grafana

---
Muthukrishnan (Muthu) Bommiah · https://muthub.org · Views are my own and do not represent my employer.