Staff Engineer - Databricks

You Will:

• Data Architecture and Design: Designing and overseeing the architecture of scalable and reliable data platforms, including data pipelines, storage solutions, and processing systems
• Data Modelling and Management: Developing and implementing data models, ensuring data quality, and establishing data governance policies
• Data Pipeline Development: Building and optimising data pipelines for ingesting, processing, and transforming large datasets from various sources
• Performance Optimisation: Identifying and resolving performance bottlenecks in data pipelines and systems, ensuring efficient data retrieval and processing
• Technology Evaluation and Innovation: Staying abreast of emerging data technologies and exploring opportunities for innovation to improve the organisation’s data infrastructure
• Troubleshooting and Problem Solving: Diagnosing and resolving complex data-related issues, ensuring the stability and reliability of the data platform
• Data Security and Compliance: Implementing data security measures, ensuring compliance with data governance policies, and protecting sensitive data
• Perform other duties as assigned

You Have:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 10+ years of experience in data engineering or a similar role.
• Enterprise SaaS software solutions with high availability and scalability
• Solution handling large scale structured and unstructured data from varied data sources
• Experience in building and maintaining data platform systems such as distributed compute, data orchestration, distributed storage, streaming infrastructure ensuring scalability, reliability, efficiency and security
• Working with Product engineering team to influence designs with data, AI and analytics use cases in mind
• In depth experience in System design involving large Petabytes of data with Databricks Lakehouse
• Experience in modern AI/Data infrastructure patterns, Semantics layer Organizing data for AI agents (metadata, context)
• AWS, GCP, Snowflake and Data pipeline frameworks like Airbyte/Airflow
• Programming languages like Python, SQL, and potentially Java or Scala
• Modern software engineering practices like Kubernetes, CI/CD, IAC tools (Preferably Terraform), Observability, monitoring and alerting
• Solution Cost Optimisations and design to cost
• Driving engineering excellence initiative
• Legally eligible to work in India on an ongoing basis
• Fluency in English is required