Sr. Data Engineer / Sr. BI Developer - DA&A

Job Summary

We are seeking a Sr. Data Engineer / Sr. BI Developer - DA&A with 4-7 years of experience in designing and building scalable cloud-based data platforms using AWS, Python, AWS Glue, Apache Airflow, Amazon S3, Amazon Redshift, and AWS Step Functions. Candidate must have relevant experience of 4 to 7 Years. The ideal candidate should have hands-on expertise in developing robust ETL/ELT pipelines, orchestrating complex workflows, implementing data ingestion frameworks, and supporting enterprise data warehousing solutions. Experience with Python, PySpark, SQL, and cloud-native data processing is essential. The role involves building and optimizing data pipelines using AWS Glue, managing workflow orchestration through Airflow and Step Functions, designing data lake architectures on Amazon S3, and supporting analytics and reporting solutions on Amazon Redshift. The candidate should be proficient in data modeling, performance tuning, monitoring, troubleshooting, CI/CD practices, and automation. Exposure to modern data engineering tools such as Snowflake, Databricks, dbt, Terraform, or similar cloud technologies will be considered an added advantage.

Key Responsibilities

• Design, develop, and maintain scalable, cloud-based data platforms and solutions on AWS.
• Build, optimize, and manage ETL/ELT data pipelines using AWS Glue, Python, and PySpark.
• Orchestrate complex data workflows and batch processes using Apache Airflow and AWS Step Functions.
• Design and implement data lake architectures on Amazon S3 to support analytics and reporting needs.
• Develop and support enterprise data warehousing solutions on Amazon Redshift, including data loading, transformations, and performance tuning.
• Implement robust data ingestion frameworks to integrate data from multiple structured and unstructured sources.
• Collaborate with data analysts, BI developers, and business stakeholders to understand data requirements and translate them into technical solutions.
• Apply data modeling best practices for analytical and reporting use cases, including dimensional and relational models.
• Monitor data pipelines and workflows, proactively identify issues, and perform root cause analysis and troubleshooting.
• Implement CI/CD practices for data engineering workflows, including version control, automated testing, and deployment automation.
• Ensure data quality, integrity, and consistency through validation, reconciliation, and automated checks.
• Optimize data processing performance and costs by leveraging appropriate AWS services and configurations.
• Create and maintain technical documentation for data pipelines, data models, and workflows.
• Support operationalization and automation of recurring data engineering tasks and processes.
• Explore and evaluate modern data engineering tools such as Snowflake, Databricks, dbt, Terraform, or similar technologies to enhance the data platform capabilities.

Other Capabilities

• Strong problem-solving and analytical skills with attention to detail.
• Ability to work in an agile, collaborative environment with cross-functional teams.
• Effective communication skills to explain technical concepts to non-technical stakeholders.
• Demonstrated ability to prioritize, manage multiple tasks, and deliver high-quality outcomes within timelines.
• Continuous learning mindset with interest in emerging data engineering tools and cloud technologies.
• Experience working in data analytics and data automation environments, including Data Engineering, Data Analytics, and Data Modelling (Technical).