Senior Microsoft Fabric Performance & Scalability Specialist

Role overview

We are seeking a hands-on Senior Microsoft Fabric Performance & Scalability Specialist with 10+ years of Data Engineering and Data Warehousing experience to optimize, scale, develop, and support an enterprise Microsoft Fabric platform supporting 1,500+ clients/tenants.

Key responsibilities:

1. Microsoft Fabric Performance & Scalability

• Optimize Microsoft Fabric performance, scalability, concurrency, and capacity utilization across Fabric Data Warehouse, Lakehouse, OneLake, Shortcuts, Pipelines, and Notebooks.
• Design and enhance the platform to efficiently support migration and ongoing processing for 1,500+ tenants/clients.
• Develop scalable processing strategies for high-volume and low-volume tenants while meeting data-load and reporting SLAs.
• Analyze Fabric capacity/CU utilization, throttling, concurrency, Query Insights, and workload performance.
• Implement performance optimization recommendations based on workload analysis.
• Conduct performance, scalability, concurrency, stress, and load testing.
• Establish measurable performance benchmarks and continuously improve platform performance.

2. Fabric Development & Data Processing

• Develop and enhance Fabric notebooks using PySpark, Python, and SQL for scalable full and incremental data processing.
• Optimize tenant grouping, batching, parallel processing, concurrency control, workload prioritization, watermark processing, retries, and failure recovery.
• Optimize Fabric Data Warehouse SQL queries, data loads, joins, table structures, and CLUSTER BY strategies.
• Develop reusable and scalable data-processing frameworks for enterprise workloads.
• Support production-grade development and optimization of Fabric-based data solutions.

3. New Data Source Development & Onboarding

• Work closely with ETL/Data Engineering teams to onboard new data sources and new sets of tables into Microsoft Fabric.
• Analyze source schemas, tables, relationships, data volumes, dependencies, primary keys, and incremental-load requirements.
• Perform required development, code changes, configuration, and testing in Microsoft Fabric for newly introduced data sources and tables.
• Develop and enhance Fabric notebooks, pipelines, SQL code, metadata/control frameworks, Lakehouse objects, and Fabric Data Warehouse tables.
• Configure and test full and incremental data-loading processes for newly onboarded tables.
• Perform unit, integration, data-validation, and performance testing before production deployment.
• Coordinate with ETL teams to validate source-to-target mappings, transformation logic, dependencies, and data completeness.
• Integrate new data sources into the established Bronze → Silver → Gold → Lakehouse/Fabric Data Warehouse architecture.
• Develop reusable and metadata-driven patterns to reduce development effort for future data-source and table onboarding.

4. Client Onboarding & Production Support

• Work directly with clients and internal teams during onboarding, migration, production cutover, and post-production support.
• Understand client data volumes, processing requirements, dependencies, and SLA expectations.
• Support client onboarding activities including full loads, incremental-load enablement, data validation, performance validation, and production cutover.
• Investigate and resolve client-reported data, performance, data-load, and reporting issues.
• Perform end-to-end root-cause analysis across source systems, Medallion layers, Fabric notebooks, pipelines, Lakehouse, Data Warehouse, and reporting workloads.
• Troubleshoot missing or duplicate data, failed or delayed loads, incremental-processing issues, performance degradation, and SLA violations.
• Implement permanent fixes and enhance shared frameworks to prevent recurring issues across tenants.

5. Production Operations & SLA Management

• Monitor production workloads and ensure data-processing and reporting SLAs are consistently achieved.
• Identify performance bottlenecks, capacity constraints, processing failures, and concurrency issues.
• Prioritize and resolve production incidents based on business impact and SLA requirements.
• Collaborate with ETL, Data Engineering, reporting, and client-facing teams to resolve complex production issues.
• Document root-cause analysis, corrective actions, and preventive measures.
• Continuously improve production stability, reliability, and operational efficiency.