Petabyte-Scale Controlled Data Movement (CDM) Archetype
Architected a petabyte-scale CDM pipeline at USAA and led its implementation as a bank-wide archetype at Wells Fargo, providing a resilient data backbone for analytics and AI/ML that moves 25 TB/hour.
Project Overview
I architected and delivered a petabyte-scale, configuration-driven Controlled Data Movement (CDM) pipeline at USAA for secure and scalable data transfers. Following its success, I brought this architectural blueprint to Wells Fargo, where I led the design and implementation of a bank-wide CDM archetype aligned with enterprise data governance standards.
The architecture implements a dual-perimeter security model to enforce strict data isolation, granular access controls (IAM and VPC SC), and perimeter-based ingress/egress rules. Sensitive data is protected through column-level encryption and Google DLP-based de-identification.
Key Features
- Achieved a 20x improvement in scalability and 10x reduction in latency at USAA.
- Features automated audit logging, data validation, and rule-based reprocessing for end-to-end observability.
- Integrates with observability tools like Datadog and ELK, BI tools like Power BI and Looker, and governance platforms like Google Dataplex and Azure Purview.
- Ensured compliance with regulatory mandates (e.g., ISCR 808-00, 809-11).
Technologies Used
Project Details
Client
USAA & Wells Fargo
Timeline
2020 - 2023
Role
Architect
More Projects
© 2025 Muthu. All rights reserved.