HPC Storage Engineer

Responsibilities:

• Own capacity, durability, availability, and performance characteristics of network volumes, local NVMe, and S3-compatible object storage.
• Tune the full I/O path: device and filesystem configuration, caching and read-ahead strategies, replication and erasure coding trade-offs, and client-side mount behavior.
• Diagnose hard performance problems end to end
• Lead capacity expansions, hardware refreshes, migrations, and rebalances without customer-visible disruption.
• Work with Runpod and partner networking teams to design and tune the network paths storage depends on: high-throughput east-west fabric, MTU and jumbo frames, congestion and flow control, multipath, and NIC/offload configuration.
• Understand and optimize RDMA/RoCE and high-speed IB/Ethernet fabrics as they apply to storage traffic.
• Work closely with network engineering on topology decisions, oversubscription ratios, and cross-region data movement.
• Write production code (Go, Python, or similar) for storage control-plane services, provisioning workflows, data movement pipelines, and monitoring
• Build against and extend APIs: our own control plane, S3-compatible interfaces, CSI drivers, Kubernetes APIs, vendor and cloud provider APIs.
• Automate the operations you'd otherwise do by hand. Manual runbooks are a starting point, not a destination.
• Treat infrastructure as code and participate fully in code review, testing, and CI.
• Instrument the storage fleet so its behavior is legible: IOPS, throughput, latency, error and retry rates, capacity utilization, and per-tenant consumption.
• Build dashboards, SLOs, and alerts that catch degradation before customers do.
• Participate in an on-call rotation for storage systems and drive blameless post-incident follow-through.

Requirements:

• <strong>8+ years in infrastructure, storage, or systems engineering, with substantial ownership of production storage at scale.</strong>
• Deep, practical experience with at least one distributed storage system — Ceph, MinIO, Lustre, GPFS/Spectrum Scale, MooseFS, WekaFS, VAST, ZFS-based systems, or comparable.
• Strong Linux internals and storage-stack knowledge: block layer, filesystems, NVMe, page cache, I/O schedulers, NFS/SMB, iSCSI/NVMe-oF.
• Building and/or operating S3-compatible object storage services.
• Solid networking fundamentals with specific experience tuning networks for storage workloads.
• Proficiency in writing and shipping production code in Go, Python, Rust, or similar (not just scripting).
• Hands-on experience with observability tooling (Prometheus, Grafana, Datadog, or equivalent) including designing the metrics, not just consuming them.
• A track record of performance analysis and debugging under real production pressure.
• Self-starting with general direction. You take a goal like "network volume read latency is hurting cold starts in EU" and come back with a diagnosis, an options analysis, and a plan without needing the work broken down for you.
• Continuous improvement. You leave systems measurably better than you found them. You notice the recurring toil, the alert that fires every Tuesday, the manual step everyone tolerates and you eliminate it.
• Ownership. You follow problems across team boundaries to resolution instead of handing them off at the edge of your component.
• Collaborative and low-ego (but high confidence).

Preferred:

• Storage for AI/ML workloads: checkpointing, dataset streaming, model weight distribution, GPU-adjacent data locality, GPUDirect Storage.
• Kubernetes storage internals: CSI drivers, PV/PVC lifecycle, StatefulSets, local persistent volumes.
• Bare-metal and colocation experience: hardware selection, vendor management, firmware, physical failure domains.
• Multi-tenant environments where isolation, fairness, and QoS are hard requirements.
• Experience in a fast-growing cloud or infrastructure provider.