Job Description
Key Responsibilities:\n- Design, build, and maintain ETL/ELT pipelines for structured and unstructured data in AWS.\n- Work with AWS services (e.g., S3, Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, Athena, DynamoDB, RDS) to manage data ingestion, transformation, and storage.\n- Ensure data quality, governance, and security across the full lifecycle.\n- Enable real-time and batch data processing to support AI-driven workflows.\n- Collaborate with AI/ML teams to prepare datasets for model training, inference, and fine-tuning.\n- Optimize data infrastructure for scalability, cost-efficiency, and performance.\n- Implement CI/CD pipelines and best practices for data engineering in a cloud-native environment.\n- Monitor and troubleshoot data pipelines to ensure high availability and reliability.\n\n### Required Skills & Qualifications:\n- 3–7 years of experience in data engineering (adjust based on seniority).\n- Strong programming skills in Python, SQL, and/or Scala/Java.\n- Expertise in AWS cloud ecosystem (S3, Glue, EMR, Redshift, Athena, Kinesis, Lambda, etc.).\n- Experience with data pipeline orchestration tools (Airflow, Step Functions, Dagster, or similar).\n- Proficiency with big data frameworks (Spark, Hadoop, or Flink).\n- Familiarity with data modeling, warehousing, and schema design.\n- Solid understanding of data governance, lineage, and security (IAM, Lake Formation, encryption). \n- Experience with real-time streaming data (Kafka, Kinesis, or equivalent).\n- Knowledge of DevOps practices (Terraform/CloudFormation, CI/CD, Git, Docker).
