Introducing SUNK Auto Repair: Safe, Automatic Recovery for Drained GPU Nodes

Healthy hardware shouldn’t wait for a human
Introducing SUNK Auto Repair: Safe, Automatic Recovery for Drained GPU Nodes

On a large GPU cluster, some of your healthiest accelerators are probably idle right now. Not broken, not busy—just parked.

When a compute node hits a problem, Slurm drains it: the node stops accepting new jobs so the scheduler can’t place work on hardware that just failed a check. That’s exactly the right move. You don’t want a thousand-GPU training run landing on a node that just tripped a health probe, hit a prolog error, or dropped a network link in the middle of a collective. Draining protects your jobs.

The trouble is what happens next. Many of the faults that trigger a drain are transient—a health check that clears on its own, a teardown that just needed a few more seconds, a brief network blip. The fault is gone in seconds. The drain is not. It sits there until a person notices the node, confirms it looks healthy, and runs scontrol update ... State=RESUME. On a quiet afternoon, that’s a few minutes of toil. On a Friday night before a long weekend, that node can sit idle for days.

At fleet scale, those parked-but-healthy nodes add up: less usable capacity on the most contended resource you own, and a steady stream of manual GPU node recovery work for your on-call team.

That’s the gap SUNK Auto Repair closes.

The lifecycle of a drained GPU node in SUNK Auto Repair. Draining stops only new work—jobs already running keep going. The recoverable, re-verified path heals automatically; everything else stays drained for a person.

What SUNK Auto Repair does 

SUNK Auto Repair is a new capability for CoreWeave SUNK, our managed Slurm-on-Kubernetes platform. It watches for drained GPU compute nodes, works out why each one was drained, re-checks that the node is actually healthy, and—when it’s safe to do so—returns the node to service automatically. No ticket, no 2 a.m. RESUME, no healthy hardware waiting on a human.

The important word there is safe. The naive version of this feature is a cron job that resumes everything every few minutes, and it’s a trap. Blindly resuming nodes is worse than leaving them idle: put a node with a genuinely bad GPU back in the pool, and the next distributed job that lands on it can crash hours in. 

An idle healthy node is a minor annoyance; a wrongly resumed broken node is an expensive, hard-to-debug failure. So the entire design of SUNK Auto Repair is built around one principle: act only when the evidence says it’s safe, and otherwise do nothing and leave the node for a human.

How SUNK Auto Repair automates GPU node recovery

SUNK Auto Repair runs a simple loop over every drained GPU node: observe, classify, verify, act.

  1. Observe
    It notices that a node has been drained and reads the drain reason Slurm recorded.
  2. Classify
    It matches that reason against a policy of known drain conditions. CoreWeave curates this policy: we’ve mapped the drain reasons we see across our fleet into the ones that are genuinely recoverable and the ones that aren’t.
  3. Verify
    For a recoverable reason, SUNK Auto Repair doesn’t just assume the fault has passed. It re-checks the node’s health at repair time, using the same HPC Verification checks CoreWeave already runs across the fleet. If the node doesn’t pass a fresh check, it stays drained.
  4. Act
    If the reason is recoverable and the node passes verification, SUNK Auto Repair returns it to service. If anything is off—an unrecognized reason, a failed health check—it holds the node and leaves it alone.

That last part is the heart of it: the system is fail-closed. Anything outside the curated safe set is left exactly where it is, drained, for a human to look at. When in doubt, SUNK Auto Repair does nothing.

The SUNK Auto Repair loop for GPU node recovery. The fail-closed default (the large panel) holds a node drained for a human whenever anything is uncertain; a node is resumed only when the reason is recoverable and a fresh health check passes.

Why SUNK Auto Repair is safe

Automated GPU node recovery that puts hardware back into a production scheduler has to earn trust. A handful of design choices keep SUNK Auto Repair conservative by construction:

  • A drain is a safety claim, not a queue to be cleared. The default answer is always “leave it alone.” A node is only ever resumed when the policy and a current health check both agree.
  • Health is re-verified at resume time. A fault that tripped a drain an hour ago tells you nothing about the node right now, so SUNK Auto Repair confirms current health before it acts, instead of assuming the problem has cleared.
  • Running jobs are never disrupted. Draining a Slurm node only stops new work from landing on it; jobs already running keep going. Returning a node to service simply makes it schedulable again — there’s nothing to interrupt.
  • Repairs are rate-limited. SUNK Auto Repair enforces a cooldown between repair attempts on the same node and a cap on how many times it will repair a node in a day. A node that keeps flapping between drained and healthy quickly hits its limit and is left drained for a person to look at, rather than bouncing in and out of the pool. These limits are managed by CoreWeave as part of the safety envelope.
  • Persistent faults stay put. Uncorrectable memory errors, a GPU that’s fallen off the bus, an unfamiliar drain reason—none of these are in the recoverable set, so the node stays drained. SUNK Auto Repair is built to hand you the genuinely hard cases, not to paper over them.
Fail-closed in practice: transient, re-verifiable faults are repaired automatically, while persistent faults and anything unfamiliar are handed to a person. Both outcomes are safe.

What’s on the roadmap for SUNK

We designed the policy layer to eventually be yours. User-defined auto-repair policies are on the roadmap: the ability to tailor which drain conditions are auto-repaired to your own workloads and risk tolerance, on top of the CoreWeave-curated baseline. That capability is coming; it isn’t available yet. 

For now, the curated policy delivers the benefit—healthy nodes back in service, automatically and safely—with no tuning required on your side.

Get started with SUNK Auto Repair

Healthy hardware shouldn’t wait for a human. With SUNK Auto Repair, it doesn’t have to.

SUNK Auto Repair is available for CoreWeave SUNK clusters. To turn it on for your fleet, reach out to your CoreWeave representative, and we’ll enable it and walk you through what to expect.

Related blogs

SUNK,
Copy code
Copied!
SUNK,