Location: San Francisco (Onsite) Type: Full-time Start Date: ASAP
What You'll Do
- Implement and tune workload management and scheduling across on-prem, cloud, and hybrid environments
- Build and manage monitoring and observability infrastructure
- Drive automation strategy for HPC and GPU deployments
- Technical engagement with customers of heterogeneous HPC and AI compute environments
- Validate high-speed interconnect fabrics and diagnose performance issues at the network, PCIe, and device level
What You Bring Deep experience across several of the following:
- Large-scale GPU cluster architecture and operations (multi-node, multi-rack GPU deployments in production)
- High-speed interconnects: InfiniBand (NDR/HDR), RoCE, RDMA design, deployment, and performance tuning
- Parallel and distributed storage systems (Lustre, CEPH, ZFS, GPFS, or comparable) at petabyte scale
- Job scheduling and resource management for GPU workloads (SLURM, PBS, Kubernetes-based, or equivalent)
- Linux systems engineering at scale, including bare-metal provisioning, kernel tuning, and hardware-level debugging
- BMC/IPMI/Redfish interfaces and hardware lifecycle management
- NVIDIA GPU stack: drivers, CUDA, NCCL, NVML, GPU health monitoring and diagnostics
Strong familiarity with:
- Virtualized and containerized GPU workloads and performance benchmarking
- Network and cluster performance validation
- Python, Bash, or Go for infrastructure automation and tooling
- Air-gapped, DDIL, or otherwise restricted deployment environments
- Vendor-diverse hardware environments
You should have:
- A track record of owning compute infrastructure end to end, from architecture through production operations
- Experience leading technical decisions on cluster design, storage architecture, or network topology
- Comfort working directly with customers and end users across a range of technical sophistication
- The ability to translate operational pain into product requirements and engineering priorities
- Interest in turning years of hard-won systems knowledge into software that makes hardware infrastructure run itself
Requirements
- 7+ years of experience in HPC systems engineering, large-scale infrastructure, or GPU compute operations
- Willingness to work startup hours, in-person (some weekends included) at our San Francisco office
- Work authorization in the United States
What we offer:
- Competitive salary
- A role where your operational experience directly shapes the product
- Problems that span GPU clusters, high-speed networking, storage, and scheduling across cloud, on-prem, and classified environments
- A team that has done this work and understands what it takes
How to Apply Email: team@cosmiclabs.io Subject line: Principal HPC Engineer / [Your Name]
Include in your email:
- Your name
- Why this role and why Cosmic Labs
- What you bring technically
- Soonest available start date
- GitHub or GitLab link (if applicable)
- Confirmation of work authorization in the U.S.
- Confirmation of willingness to work full-time, in-person in San Francisco
Attach: PDF resume