Senior ML Systems Engineer, Inference

Responsibilities

• Define how we measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build the tooling that makes those measurements rigorous and repeatable.

• Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect.

• Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments.

• Turn what you learn into production-ready runtimes, configurations, and defaults that customers benefit from automatically.

• Work closely with product and infrastructure teams to shape how inference is offered on Runpod.

• Keep up with the fast-moving inference ecosystem, including the open-source community, and decide what's worth adopting, what's worth building, and what's worth contributing back.

• Trace bottlenecks in the serving engine/runtime and implement fixes when configuration tuning is not enough.

Requirements

• 5+ years of professional system engineering experience.

• Deep, hands-on experience with vLLM, SGLang (or a comparable serving engine) in production or at serious benchmark scale.

• Strong software engineering skills in Python. You're comfortable working in large, performance-critical codebases.

• A solid understanding of what drives LLM inference performance: batching, memory, parallelism, and the trade-offs between latency and throughput.

• Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving.

• Rigor in benchmarking and performance analysis, plus comfort with GPU profiling tools.

• The ability to explain your results clearly in writing and turn them into decisions.

Preferred

• Experience writing or tuning GPU kernels in CUDA or Triton.

• Contributions to inference or ML systems projects.

• Experience with multi-node GPU systems and high-speed networking.

• Experience at a company where inference cost and latency were core business metrics.