Principal Software Development Engineer (Microservices)

Role

We are looking for a Principal Software Development Engineer (Microservices) to join our team. This is a hybrid role (in Santa Clara, CA office, minimum 3 days per week), reporting to the Senior Director, Software Development Engineering in the Engineering ZIA Core department. You will work on the control plane for the Zscaler Internet Access (ZIA) product, helping evolve the platform to meet massive scale with cloud-native, microservices-based architecture.

What you’ll do (Role Expectations)

• Lead the technical roadmap to modernize the ZIA control plane using API-first, domain-aligned microservices with clear service ownership and multi-region architectures (active-active, failover, disaster recovery)
• Design, develop, and operate low-latency, event-driven services end to end (API through storage) utilizing resilience patterns, progressive delivery (blue/green, canary, feature flags), GitOps workflows (Argo CD/Flux), and safe rollout/rollback strategies
• Drive engineering excellence through standards, code reviews, testing, paved-road patterns, API and event contract governance (versioning, consumer-driven contracts, schema registry, DLQs), and cross-functional partnerships across Product, SRE, Security, Data, and Architecture while representing designs in customer/partner forums
• Own service reliability and operability end to end, including SLI/SLOs (p95/p99 latency, availability), SLO-based alerting, error budgets, incident response, postmortems, and advanced observability with distributed tracing (OpenTelemetry, Jaeger/Tempo) and structured logging
• Embed microservices security (OAuth2/OIDC, mTLS, service-to-service authZ, KMS/Vault), supply-chain integrity (SBOM, SLSA), performance optimization, capacity planning, and cloud spend/autoscaling across AWS primitives

Who You Are (Success Profile)

• You thrive in ambiguity and are comfortable building the path as you walk it, viewing unstructured environments as raw material to build something meaningful.
• You act like an owner with a passion for the mission, operating with integrity, a bias for action, and the agility to navigate between high-level strategy and hands-on execution.
• You are a problem-solver who is energized by complex technical challenges and driven to deliver high-impact solutions.
• You are a high-trust collaborator who builds strong team relationships, embracing continuous feedback, candor, and respect to earn trust and achieve shared goals.
• You are a learner with a growth mindset, constantly seeking feedback and developing your skills with purpose to become a stronger teammate.

What We’re Looking for (Minimum Qualifications)

• Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain
• 12+ years in software development with 4+ years building and operating production microservices at significant scale, including proficiency in Go
• Deep experience with distributed systems and messaging (Kafka preferred; SQS/RabbitMQ acceptable), alongside patterns such as CQRS and Event Sourcing where fit-for-purpose
• Kubernetes in AWS at scale (EKS preferred), strong Docker fundamentals, Infrastructure-as-Code, and GitOps workflows
• Strong SQL (PostgreSQL/MySQL/Amazon RDS/Aurora) and NoSQL (Cassandra/MongoDB/DynamoDB) design and scaling with familiarity with multi-region data trade-offs, alongside performance engineering and cost optimization in AWS
• Comprehensive reliability and security experience including SLIs/SLOs, error budgets, metrics (Prometheus), tracing (OpenTelemetry), SLO-based alerting, incident management, postmortems, service mesh (Istio/Linkerd), mTLS, OAuth2/OIDC, and secrets management (KMS/Vault)

What Will Make You Stand Out (Preferred Qualifications)

• Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
• Chaos engineering, advanced load/performance testing, fault injection, and disaster recovery drills combined with multi-region design expertise (active-active, partitioning, replication, eventual consistency) and migration leadership (strangler patterns, domain decomposition)
• Event contract governance (Avro/Protobuf with schema registry), DLQ processing strategies, and data replay tooling
• OSS contributions, technical talks, patents, or published design writing