Big Data Engineer

We are looking for a <strong>Big Data Engineer</strong> to design, build, and operate large-scale data pipelines and analytical infrastructure that transform high-volume raw data into reliable, query-ready datasets for analytics, reporting, and data-driven products.\nOur data platform ingests and processes data from multiple sources and serves analytics, data science, product, and downstream applications. In this role, you will own data pipelines end-to-end—from ingestion and transformation to warehousing, orchestration, data quality, and observability.\nA key part of the role is owning <strong>ClickHouse as our primary analytical data store</strong>. You will be responsible for designing scalable data models, optimizing query performance, and ensuring the platform remains reliable and cost-efficient as data volumes and workloads grow.\nYou will work closely with <strong>data scientists, analysts, product engineers, and other engineering teams</strong> to build a modern, cloud-native data platform.\n### What You'll Do\n- Design, build, and maintain robust <strong>batch and streaming data pipelines</strong> that ingest data from multiple sources into analytical data stores.\n- Build and operate <strong>Apache Airflow DAGs</strong>, including scheduling, dependencies, retries, backfills, idempotency, concurrency, and failure handling.\n- Develop analytics-ready datasets using <strong>dbt</strong>, following well-structured staging, intermediate, and mart layers with appropriate tests, documentation, and incremental models.\n- Own <strong>ClickHouse</strong> as the primary analytical store, including:\n - Schema and table design using the MergeTree family of engines\n - Partitioning and sorting/primary key strategies\n - Materialized views\n - Distributed and replicated table architectures\n - Query and memory optimization\n - High-volume data ingestion and performance tuning\n- Work with <strong>BigQuery</strong> where cloud data-warehouse patterns are appropriate, including data modeling and query/cost optimization.\n- Design and operate <strong>NoSQL and key-value data stores</strong>, including Bigtable, DynamoDB, and Redis, based on specific access patterns and performance requirements.\n- Build and maintain <strong>data-quality frameworks</strong> covering validation, testing, freshness, completeness, reconciliation, and anomaly detection.\n- Implement <strong>monitoring, alerting, structured logging, and observability</strong> for data pipelines and services.\n- Own pipeline SLAs, incident response, troubleshooting, and root-cause analysis.\n- Manage <strong>backfills, safe re-runs, schema evolution, and data migrations</strong> while minimizing downstream impact.\n- Build reproducible, containerized environments using <strong>Docker</strong> and contribute to <strong>CI/CD and Infrastructure as Code</strong> practices.\n- Partner with analysts, data scientists, product managers, and product engineers to translate business and technical requirements into scalable data models and pipelines.\n- Continuously improve pipeline reliability, scalability, performance, and infrastructure cost efficiency.\n### Must-Have Requirements\n- <strong>4-6 years of experience</strong> in data engineering or a closely related field, with strong hands-on production experience.\n- Expert-level <strong>SQL</strong> and strong <strong>Python</strong> skills, with experience writing production-grade, maintainable, and well-tested code.\n- Strong hands-on experience with <strong>Apache Airflow</strong> or a comparable workflow orchestration platform, including DAG design, scheduling, retries, backfills, dependency management, idempotency, and concurrency.\n- Hands-on experience with <strong>dbt</strong> or a comparable transformation/ELT framework, including modular models, testing, source management, documentation, and incremental processing.\n- <strong>Expert-level production experience with ClickHouse</strong>. This is a core requirement and should include:\n -MergeTree engine family\n -Partitioning and primary/sorting keys\n -Materialized views\n -Distributed and replicated tables\n -Query and memory optimization\n -High-volume ingestion and performance tuning\n- Production experience with a <strong>cloud-based columnar/OLAP warehouse</strong>, such as BigQuery, including data modeling and performance/cost optimization.\n- Hands-on experience with <strong>NoSQL and key-value databases</strong>, such as Bigtable, DynamoDB, and Redis, including data modeling, partition/key design, access patterns, and caching strategies.\n- Strong understanding of <strong>ETL/ELT and dimensional/layered data modeling</strong> principles.\n- Experience with <strong>GCP and/or AWS</strong> and familiarity with <strong>Docker, Git, and CI/CD</strong>.\n- Strong focus on <strong>data quality, reliability, observability, and correctness</strong>.\n- Strong ownership, problem-solving, and communication skills, with the ability to collaborate effectively across engineering, analytics, data science, and product teams.\n### Nice-to-Have\n- Experience with <strong>search platforms</strong> such as Elasticsearch, OpenSearch, Apache Solr, or Vespa, including indexing pipelines, schema design, and relevance/performance tuning.\n- Experience with <strong>Aerospike</strong> or other high-performance, low-latency distributed key-value/NoSQL systems.\n- Experience building <strong>streaming and event-driven pipelines</strong> using Kafka, Pub/Sub, or similar technologies.\n- Experience with <strong>Change Data Capture (CDC)</strong> patterns and technologies.\n- Experience with <strong>Apache Spark</strong> and data-lake architectures using object storage such as GCS or S3.\n- Experience with <strong>Terraform</strong> or other Infrastructure as Code tools and <strong>Kubernetes</strong>.\n- Experience with data-quality and observability tools such as <strong>Great Expectations, Soda, Monte Carlo, or advanced dbt testing</strong>.\n- Understanding of <strong>data platform cost optimization / FinOps</strong> practices.\n- Experience handling <strong>high-volume e-commerce, product catalog, behavioral, or event data</strong>.\n- Familiarity with a second backend programming language, particularly <strong>Go</strong>.\n### What Success Looks Like\n- Data pipelines are <strong>reliable, well-tested, observable, and consistently meet freshness and completeness SLAs</strong>.\n- Data models are <strong>clean, scalable, documented, and trusted</strong> by analytics, data science, and product teams.\n- Data-quality issues are identified <strong>before they impact downstream consumers</strong>.\n- Pipeline failures are diagnosed and resolved quickly, with clear root-cause analysis and preventive actions.\n- ClickHouse and the broader analytical platform <strong>scale smoothly with growing data volumes and query workloads</strong>.\n- Data infrastructure remains <strong>performant and cost-efficient</strong> as usage grows.\n- Downstream teams can confidently rely on the platform for <strong>analytics, reporting, experimentation, and data-driven product experiences</strong>.