Senior Cluster Site Reliability Engineer (SRE)

About

Voleon is a technology company applying state-of-the-art machine learning techniques to real-world finance problems. We have led the industry for nearly two decades, becoming a multibillion-dollar asset manager with ambitious future goals.

As a Senior Cluster Site Reliability Engineer (SRE), you will help scale our research compute cluster, ensuring high uptime, reliability, and robustness. You will support on-prem and cloud infrastructure, leveraging IaC, automation, and SRE principles to maintain a 24/7 HPC platform for cutting-edge machine learning research.

The Cluster Operations team triages and mitigates real-time operational issues, engineering systemic improvements and architectural fixes. You will collaborate with engineering teams to improve monitoring and operational frameworks to meet rigorous SLAs.

Responsibilities

• Be a first responder for cluster outages or issues, triaging and resolving urgent problems.
• Ensure high cluster uptime (multiple nines) and define and track SLAs.
• Diagnose recurring problems and engineer precision solutions with engineering teams.
• Develop robust metrics and observability for cluster health; build custom mechanisms when needed.
• Help design and enforce fair cluster usage policies.
• Assist in forecasting cluster growth and selecting scale-up strategies.
• Optimize operations for cost and usability.

Requirements

• 5+ years in SRE or DevOps roles, preferably as a senior engineer or tech lead.
• Knowledge of HPC/batch compute frameworks (Slurm, Kueue, AWS/GCP Batch) and/or ML training systems (Kubeflow, MLflow, Horovod).
• Ability to develop scripts/utilities in scripting languages (Python, Ruby, etc.).
• Familiarity with infrastructure-as-code and configuration management tools (Terraform, Ansible).
• Experience with cloud infrastructure (AWS or GCP).
• Familiarity with observability stacks (Prometheus, Grafana, Loki, ELK, OpenTelemetry).
• Experience with distributed storage (Lustre, Ceph, S3).
• System engineer mindset leveraging automation.
• Bachelor degree in computer science.

Preferred Qualifications

• Hands-on experience with HPC frameworks (Slurm, Grid Engine) and Kubernetes-based job orchestrators (Airflow, Kueue, Kubeflow Pipelines).
• Familiarity with ML frameworks (PyTorch/Tensorflow, JAX, Horovod, DeepSpeed).
• Familiarity with hybrid/on-prem environments.
• Experience with containerization (Docker, Podman, Singularity) for HPC/batch compute.
• Experience with HPC networking (InfiniBand, RDMA).
• Solid security/IAM foundations (Identity management, AWS/GCP IAM, Zero Trust).

Benefits

• Medical insurance
• Dental insurance
• Vision insurance
• Life and AD&D insurance
• 20 days paid time off
• 9 sick days
• 401(k) plan with company match

Equal Opportunity Employer

The Voleon Group is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by law.