Senior Software Engineer (Inference)

Join our team as a Senior Software Engineer (Inference) where you will lead designs, raise engineering standards, and deliver measurable improvements to latency, throughput, and reliability across multiple services. You will partner with product, orchestration, and hardware teams to evolve our Kubernetes-native inference platform and meet strict P99 SLAs at scale. Additionally, you will mentor junior engineers, review cross-team designs, and elevate coding/testing standards.

• Lead the design and architecture of the Kubernetes-native inference platform, ensuring it meets strict P99 SLAs at scale.
• Implement advanced optimizations and quantifying their impact on latency, throughput, and reliability across multiple services.
• Mentor junior engineers, review cross-team designs, and elevate coding/testing standards within the team.

• Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry)
• Practical knowledge of inference internals: batching, caching, mixed precision (BF16/FP8), streaming token delivery
• Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems and performance
• Proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work
• IC3: ~3–5 years; IC4: ~5–8 years industry experience building distributed systems or cloud services
• This position requires access to export controlled information. To conform to U.S. Government export regulations applicable to that information, applicant must either be (A) a U.S. person, defined as a (i) U.S. citizen or national, (ii) U.S. lawful permanent resident (green card holder), (iii) refugee under 8 U.S.C. § 1157, or (iv) asylee under 8 U.S.C. § 1158, (B) eligible to access the export controlled information without a required export authorization, or (C) eligible and reasonably likely to obtain the required export authorization from the applicable U.S. government agency
• Contributions to inference frameworks (vLLM, Triton, TensorRT-LLM, Ray Serve, TorchServe)
• Leading multi-team initiatives or partnering with customers on mission-critical launches
• Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies