Data Engineer

ABOUT THE ROLE

We are seeking a Principal Data Engineer to serve as the most senior technical authority in our data and analytics practice. This is a hands-on leadership role: you will own the architecture and engineering standards for AWS-native data platforms, lead and mentor multi-pod engineering teams, and act as the trusted technical counterpart to customer stakeholders.

KEY RESPONSIBILITIES

ARCHITECTURE & TECHNICAL DESIGN

• Own end-to-end architecture for AWS-native data platforms — lakehouse, warehouse, streaming, and data product layers — from discovery through production hardening.
• Produce and maintain architecture artefacts: solution design documents, HLD/LLD, data flow and lineage diagrams, ADRs (Architecture Decision Records), and reference implementations.
• Define the target-state roadmap and lead modernisation and migration programmes (on-premises Hadoop/Teradata/Informatica → AWS; legacy ETL → Glue/EMR/Iceberg).
• Evaluate and select services, table formats, and third-party tooling with clear trade-off analysis on cost, performance, operability, and lock-in.
• Design for non-functional requirements from day one — availability, recoverability (RTO/RPO), scalability, multi-tenancy, and disaster recovery.

PLATFORM & PIPELINE ENGINEERING

• Direct the design and build of scalable batch and streaming pipelines using AWS Glue, Amazon EMR, Amazon Kinesis, Amazon MSK, and AWS Lambda.
• Model and optimise analytical data stores in Amazon Redshift and S3-based data lakes — partitioning strategy, file formats, compaction, distribution and sort keys, workload management.
• Set the standards for CDC and ingestion patterns from operational databases and SaaS sources using AWS DMS, Glue connectors, and Kinesis.
• Own orchestration patterns across Amazon MWAA, AWS Step Functions, and Amazon EventBridge, including idempotency, retry, backfill, and SLA-breach handling.
• Own data cataloguing, lineage, and fine-grained access control via AWS Glue Data Catalog and AWS Lake Formation.
• Define the data quality and observability framework — contracts, expectations, freshness and volume checks, reconciliation, and alerting.

PERFORMANCE TUNING & COST OPTIMISATION

• Lead deep performance engineering across Redshift, Athena, Spark on EMR/Glue, and streaming workloads: query plan analysis, skew and spill remediation, partition pruning, caching, concurrency scaling, and shuffle optimisation.
• Establish benchmarking and profiling practice — define baselines, instrument workloads, and drive measurable improvements in latency, throughput, and job runtime.
• Own the FinOps posture for the data estate: right-sizing, Spot and Graviton adoption, storage tiering and lifecycle policies, Redshift RA3/serverless sizing, and per-workload cost attribution and chargeback.
• Set and enforce cost and performance SLOs, and run regular optimisation reviews with engineering and finance stakeholders.

ENGINEERING STANDARDS & CODE REVIEW

• Define and enforce engineering standards: coding conventions, repository structure, branching strategy, testing pyramid, documentation, and definition of done.
• Act as final reviewer and approver on critical pull requests; run structured code review sessions and raise the review bar across the team.
• Own CI/CD for data pipelines — automated testing (unit, integration, data quality), linting, security scanning, environment promotion, and release management.
• Champion infrastructure as code (Terraform, AWS CDK, or CloudFormation) and reusable, modular, well-tested platform components over bespoke one-off builds.
• Drive technical debt visibility and remediation planning alongside feature delivery.

TEAM LEADERSHIP & MENTORING

• Lead and technically line-manage a team of data engineers across multiple squads; allocate work, set technical goals, and own delivery quality.
• Mentor and coach senior and mid-level engineers; run design clinics, brown-bag sessions, and structured upskilling and certification paths.
• Contribute to hiring — technical screening, interview panel design, and calibration of the evaluation bar.
• Provide input to performance reviews, career development conversations, and succession planning for key technical roles.
• Build a culture of ownership, documentation, and blameless post-incident learning.

STAKEHOLDER MANAGEMENT & COMMUNICATION

• Act as the senior technical point of contact for customer architects, data leaders, and business sponsors; translate business objectives into technical roadmaps and vice versa.
• Present architecture, trade-offs, risks, and cost implications to both engineering audiences and CxO-level stakeholders with equal clarity.
• Manage expectations on scope, sequencing, and delivery risk; escalate early with options rather than problems.
• Support pre-sales and solutioning — effort estimation, technical proposals, solution walkthroughs, and proof-of-concept design.
• Partner with analysts, data scientists, and product owners to shape production-grade, consumable data products.

GOVERNANCE, SECURITY & COMPLIANCE

• Embed security controls into every pipeline — encryption at rest and in transit, KMS key management, IAM least privilege, VPC and network isolation, secrets management.
• Own PII discovery, classification, masking, tokenisation, and retention patterns; ensure designs meet applicable regulatory obligations (India DPDP Act 2023, GDPR where relevant) and audit requirements under ISO 27001 and SOC 2.
• Define data governance operating model in partnership with security and compliance functions — data ownership, access request workflows, and audit logging via AWS CloudTrail and Lake Formation.