Job Details

ML Infrastructure Engineer

We are building AI physics models that don't just predict, but understand cause and effect within climate.

What You'll Do

• Architect and operate distributed training clusters (e.g. 12+ nodes, 8 GPUs per node) using GKE, Kubernetes, and cloud-native infra
• Design scalable, efficient data pipelines for petabyte-scale datasets
• Implement and optimize model/data/pipeline parallelism across foundation models
• Deploy, monitor, and debug large-scale multi-node GPU training jobs using DDP, FSDP, and DeepSpeed
• Tune low-level system components (e.g. CUDA, NCCL, network interfaces) for max throughput
• Build cluster observability tools: failure detection, logging, monitoring, and autoscaling
• Collaborate with research and modeling teams to productionize experiments at scale

You'd Be Great If You

• Have deep hands-on experience with distributed training frameworks (FSDP, DeepSpeed, DDP)
• Know how to set up and debug Kubernetes/GKE GPU clusters, from CUDA to networking
• Are fluent in PyTorch and familiar with its performance quirks (e.g., dataset loading, sampler design)
• Have worked on ML infra at scale (multi-node, multi-GPU setups, 100B+ param models)
• Understand sampling techniques, data sharding, and performance tuning across the ML stack
• Can spot a NCCL timeout from a mile away and know how to fix it
• Value rapid iteration, ownership, and scaling up ambitious systems with a lean team