Automation Test Engineer

Overview of Role

As a Senior QA Engineer – Data, you will lead the QA effort for complex data engineering pipelines on Google Cloud Platform. You’ll design and implement multi-layered testing strategies—integration, end-to-end, and data quality tests—across tools like dbt, Dataflow, Dataproc, BigQuery, AlloyDB, Cloud SQL, Cloud composer and Cloud Run, etc. Your work ensures the accuracy, reliability, and performance of data systems at scale.

Responsibilities

Testing Strategy & Test Design

• Define and maintain testing methodologies for the full GCP data engineering stack: Dataflow (TestPipeline), Dataproc (spark-testing-base or pytest), Cloud Run container tests, and SQL-based data validation in BigQuery/dbt.
• Develop and execute data quality frameworks using dbt tests (schema, singular, freshness) and external tools like Great Expectations, Soda Core, and Dataplex.

Pipeline & Database Testing

• Implement integration, and regression tests for ETL/ELT pipelines, including container- level and HTTP-triggered tests for Cloud Run.
• Use emulators or dedicated test instances to test Spanner, Cloud SQL, and AlloyDB. Validate stored procedures and database functions with sample data.

End-to-End (E2E) Pipeline Validation

• Orchestrate comprehensive E2E tests via Cloud Composer/Apache Airflow or scripting. Simulate real-world data flows and validate intermediate and final outputs.

CI/CD & Automation

• Embed QA in CI/CD pipelines (e.g., GitLab CI, Jenkins, GitHub Actions), automating test execution at all levels including data quality validations and dbt runs.
• Use IaC tools (Terraform, Deployment Manager) to provision reproducible test environments.

Collaboration & Stakeholder Engagement

• Partner with data engineers and stakeholders to review design for testability.
• Mentor junior QA team members, champion QA best practices, and lead efforts to improve data quality KPIs and test effectiveness.

Monitoring & Observability

• Utilize Cloud Logging, Monitoring, and observability tools (e.g., Elementary Data) to track pipeline health, test results, and identify anomalies.