Senior Big Data Engineer

Key Responsibilities:

• Modernize legacy, non-scalable architectures and define a scalable target-state platform
• Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables (DLT), Delta Sharing and Workflows (LakeFlow, LakeBase)
• Design and build scalable, reliable and production-ready ETL/ELT pipelines using PySpark, SQL and Databricks notebooks to ingest and transform data from diverse sources.
• Create and manage workflows using Databricks Workflows (Jobs) or orchestration tools to automate pipelines and dependencies
• Optimize Spark jobs for performance, scalability and cost efficiency (partitioning, caching, query tuning, cluster optimization)
• Implement data quality checks (e.g., DLT Expectations) and enforce governance via Unity Catalog (access control, PII masking, lineage)
• Design and implement event-driven and streaming pipelines (Kafka or equivalent)
• Ensure high data reliability through monitoring, observability and alerting

Requirements:

• BE/MS in Computer Science or a related field with 8+ years of experience in data engineering
• Strong experience in ETL/ELT pipelines, data modeling, and distributed data systems, with hands-on expertise in Databricks
• Deep proficiency in PySpark, including performance optimization, job orchestration, and large-scale data processing
• Good understanding of event streaming systems such as Kafka or equivalent technologies
• Experience working with Azure data ecosystem (ADLS, Azure services, AHDS or similar data platforms)
• Strong foundation in event-driven architecture and scalable distributed systems
• Proven ability to design, review, and clearly articulate system architecture
• Experience leveraging AI-assisted development tools (e.g., Claude, Antigravity, etc.) to improve productivity
• Solid experience in Agile delivery, estimation, and program execution
• Experience working with global customers (US/EU) in a client-facing role
• Excellent written and verbal communication skills across engineering, business, and customer stakeholders
• Strong analytical thinking and structured problem-solving ability
• High ownership, reliability, and execution focus; consistently delivers despite constraints
• Strong attention to detail while maintaining a clear big-picture perspective

Good to have skills:

• Experience in Healthcare, EHR/EMR data migration, Clinical Trials, or Life Sciences domains (at least one)
• Exposure to handling large-scale EMR/EHR integrations and reducing technical complexity across multiple data sources
• Experience working with healthcare data standards such as CDA, FHIR, and HL7, including data normalization into modern data models
• Ability to build and manage scalable data pipelines for ingestion, transformation (FHIR), data quality, governance, and near real-time processing
• Understanding of interoperability challenges across diverse healthcare systems and approaches to solve them
• Experience building patient-centric data platforms (e.g., Patient 360, Master Patient Index)
• Familiarity with data privacy, security, and compliance standards such as HIPAA

Benefits/Culture @ Digit88:

• Comprehensive insurance coverage (Life, Health, Accident, parents and in-laws are optional)
• Flexible work model focused on outcomes
• Accelerated learning with non-linear growth opportunities
• Flat organization with high ownership & accountability
• Opportunity to work on cutting-edge AI and SaaS products with global customers (primarily North America, Australia, EU and UAE)
• Accomplished Global Peers - Working with some of the best engineers/professionals globally from the likes of Apple, Amazon, IBM Research, Adobe and other innovative product companies
• Direct exposure to building and scaling real-world systems across Conversational AI, Energy/Utilities, ESG, HealthTech, IoT and more
• High-impact roles with the ability to influence product, architecture and business outcomes globally
• Learn from a founding team of serial entrepreneurs with multiple exits - high growth, high ownership and real challenges
• This is an exciting time to join Digit88 - build, scale and grow with us as part of our journey!