Senior Site Reliability Engineer - Fleet

You’ll Do\n- Build and operate monitoring and alerting for cluster health — fabric, GPU, power/thermal, and job-level signals — to detect and respond to issues proactively\n- Remotely deploy and configure large-scale HPC clusters for AI workloads using automation wherever possible\n- Automate cluster lifecycle: operating systems, firmware, drivers, and networking, managed as code (Ansible, Terraform) rather than by hand\n- Create runbooks and automated remediations for common cluster failure modes, designed so Support and HPC Support can run them safely\n- Troubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power — working closely with on-site deployment teams\n- Participate in on-call rotations and lead incident response for cluster-level problems\n- Contribute to and maintain Standard Operating Procedures, and feed clear requirements back to other engineering teams on simplification, stability, and operational efficiency\n\n### You\n- 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role\n- Have a strong understanding of modern AI infrastructure, from GPU architectures to hardware performance optimization\n- Strong understanding of Linux-based systems in a distributed environment\n- Are experienced configuring and troubleshooting InfiniBand (IB), RoCE, CLOS fabrics, 100GbE, Ethernet/switching, GPU-direct, and NCCL environments\n- Solid understanding of Python and Go, with experience working with SWE teams to improve internal tooling.\n- Experience with monitoring and alerting tools (e.g., Prometheus, Grafana, Clickhouse)\n- Proficiency in automation and configuration management tools (e.g., Ansible, Terraform)\n- Have excellent problem-solving and troubleshooting skills and an innate attention to detail\n- Passion for continuous improvement and innovation\n\n### Nice to Have\n- Experience with machine learning / deep learning frameworks (PyTorch, TensorFlow) and benchmarking tools (DeepSpeed, MLPerf)\n- Knowledge of containerization and orchestration technologies (e.g., Docker, Kubernetes)\n- Experience building and/or operating HPC resources.\n- Depth in the NVIDIA hardware and firmware ecosystem\n- Experience with data center power and thermal design\n- Background in chaos engineering or similar reliability testing methodologies\n- Understanding of compliance frameworks (SOC 2, ISO 27001, etc.)\n