Data Engineer

Job Description

Job Description – Data Engineer (AWS + Databricks)

Experience

Experience: 5+ Years

Key Responsibilities:

• Build scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
• Design and implement Medallion Architecture (Bronze, Silver, Gold) for Lakehouse.
• Develop batch and real-time pipelines using Spark Streaming, AWS Kinesis, or Glue Streaming.
• Manage data ingestion from diverse sources – RDBMS, APIs, S3, Kafka.
• Perform data profiling, validation, and quality checks.
• Optimize Spark jobs, partitions, and cluster configurations for performance.
• Implement and manage data infrastructure using AWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR.
• Set up CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).
• Collaborate with analysts and data scientists to deliver analytical datasets.

Technical Skills:

• Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Job Workflows.
• AWS: S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch.
• Programming: Python (pandas), SQL (window functions, optimization).
• IaC & DevOps: Terraform / CloudFormation, Git, Jenkins, GitHub Actions.

Performance Tuning:

• Spark optimizations, Z-ordering, partitioning, caching.