Databricks Engineer

Role: Databricks Engineer\nExperience: 4–8 years\nLocation: As per business requirement\nEmployment Type: Full-time\nJob Summary: We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.\nKey Responsibilities: Design, develop, and maintain scalable data pipelines using Databricks. Build ETL/ELT workflows for batch and streaming data processing. Develop solutions using PySpark, Spark SQL, and Delta Lake. Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation. Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms. Optimize Spark jobs for performance, scalability, and cost efficiency. Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders. Implement CI/CD pipelines and deployment automation for Databricks workloads. Ensure data quality, security, governance, and compliance. Monitor, troubleshoot, and optimize production data pipelines. Document technical solutions and follow engineering best practices.\nRequired Skills: Core Technologies – Databricks Lakehouse Platform, Apache Spark, PySpark, Spark SQL, Delta Lake, Python, SQL. Cloud Platforms (one or more) – Microsoft Azure (preferred), AWS, Google Cloud Platform. Azure Technologies (Preferred) – Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure Key Vault, Azure DevOps. Data Engineering – Data Warehousing, Data Modeling, ETL/ELT Development, Batch Processing, Streaming (Kafka/Event Hubs), Data Lake Architecture. DevOps & Version Control – Git, Azure DevOps / GitHub, CI/CD Pipelines. Preferred Qualifications – Unity Catalog, Databricks Workflows and Jobs, Delta Live Tables (DLT), MLflow, Data governance and security best practices, Infrastructure as Code (Terraform). Educational Qualification – Bachelor's or Master's degree in CS/IT/Engineering. Certifications – Databricks Data Engineer, Azure Data Engineer, AZ/900, etc. Good to Have – Real-time analytics, Lakehouse architecture, Agile/Scrum, communication and stakeholder management.