Data Engineering Modernization Architect

Role Overview:

12+ years of overall experience in data engineering, data platforms, analytics, or enterprise data architecture, with 5+ years in data platform modernization/cloud data engineering and strong hands-on expertise in Databricks, Microsoft Fabric and Azure Data Factory.

Key Responsibilities:

• Lead modernization of legacy data platforms, ETL/ELT processes and data warehouses into modern cloud-based architectures. Assess existing data platforms, pipelines, databases, ETL jobs, data models and workloads. Define As-Is, To-Be and transition architectures, Develop modernization roadmaps, Identify opportunities to simplify and consolidate legacy data pipelines, Define reusable modernization patterns and frameworks.

Key Skills Requirement:

• Strong hands-on and architectural expertise in Databricks Lakehouse architecture, PySpark, Delta Lake, Delta Live Tables / Lakeflow capabilities, Unity Catalog, Workflows / Jobs, Databricks SQL, Change Data Capture, Schema evolution, Databricks security and governance, Azure Data Factory ADF pipelines, Copy activities, Mapping Data Flows, Parameters and variables, Metadata-driven pipelines, Dynamic pipelines, Integration Runtime, Self-hosted Integration Runtime, Trigger mechanisms, Error handling, Retry frameworks, Dependency management, Pipeline monitoring, CI/CD
• The candidate should be strong in Enterprise Data Architecture, Modern Data Lakehouse, Data Warehouse, Data Lake, Lakehouse architecture, Medallion architecture, Dimensional modelling, Star/Snowflake schemas, Data Vault, Data mesh concepts, Batch and streaming architecture, CDC architecture, Event-driven data pipelines, Master/reference data, Data governance, Data lineage, Data quality, Metadata management
• Strong experience in developing reusable frameworks for Metadata-driven ingestion, Dynamic pipeline generation, Automated schema discovery, Automated table creation, Incremental load framework, CDC framework, Data quality framework, Pipeline monitoring, Exception handling, Automated reconciliation, Data validation, Deployment automation, Environment provisioning
• Strong experience in Legacy ETL modernization, Data warehouse migration, Data lake migration, ETL → ELT transformation, On-premises → Azure migration, Legacy database → Lakehouse migration, Batch → near-real-time modernization, Stored procedure modernization, Legacy ETL code conversion.