When you’re training across hundreds of GPUs, failures are inevitable. The sheer scale means that the probability of a single component breaking increases enough to become an inevitable nuisance. And to make it even more of a drain on your resources, the root of the failure is often a mystery. 

Did the error occur at the application level or is it due to the infrastructure? Was it really an NCCL communication error, or is that just where my application failed when a GPU went bad?

The difference between a hiccup and a halted training run comes down to how well your infrastructure can recover. Systems that recover quickly, checkpoint efficiently, and diagnose failures automatically deliver higher throughput and lower costs over time.

This post outlines four practical tips to validate and improve your AI infrastructure’s resilience. We draw on lessons learned from CoreWeave’s large-scale benchmarking and conversations with our industry-leading customers who are running some of the largest training workloads to date.

Quick aside: You can also explore these topics in our recent webinar, How to Maximize Resiliency with AI-Native Observability. In that session, you’ll learn the importance of AI observability and CoreWeave’s AI-native solution for boosting infrastructure visibility and reliability.  

Tip #1: Build flexibility into your training code

When your job runs across 400 GPUs, one bad node shouldn’t take the job down. Don’t ever just assume stability; instead, design for failures and reproducibility. Tips:

• Write applications that can tolerate and recover from node failures through checkpointing or other mechanisms.
• Containerize training environments (Docker, Kubernetes) to ensure reproducibility and consistent dependency resolution.
• Use orchestration systems, such as Slurm, to handle job resubmission automatically.
• Don’t overconstrain your parallelism. Make sure that jobs that can be run independently are notated in your code; otherwise, the API will assume all jobs are connected.
  

Tip #2: Instrument everything from GPU health to application logic

You can’t recover from what you can’t see. Nine times out of ten, what looks like a network error is actually your application misbehaving.

Instrumentation turns blind guesses into accurate diagnoses. A full-suite observability stack unifies infrastructure and training metrics to capture telemetry and potential errors before they become big problems. 

For example, imagine a slowdown caused by thermal throttling. With proper instrumentation, you can see a drop in Model Flop Utilization (MFU), which measures how efficiently GPUs perform, correlate directly with a temperature spike in that rack.

Tips:

• Implement fine-grained instrumentation across hardware and software layers: GPU utilization, network bandwidth, and job-specific telemetry.
• Differentiate infrastructure failures (e.g., degraded NIC, thermal throttling) from application errors (e.g., tensor shape mismatch).
• Integrate observability tools like Grafana and Weights & Biases to correlate MFU drops with system-level metrics.
  

Tip #3: Automate alerts and recovery

When scaling to hundreds of GPUs, automation isn’t optional. If you are trying to figure out which node failed in a 400-node job, you can’t scour the metrics from every GPU. Automation is what turns an outage into a momentary blip. 

For example, internal testing at CoreWeave has shown that automated recovery can be approximately 3x faster than manual intervention, helping sustain Effective Training Time Ratios (ETTR), the percentage of total training time that GPUs spend doing productive work, above 96%.

Tips:

• Implement automated health checks and “hang detectors.”
• Use heartbeat monitors (e.g., PyTorch’s TORCH_NCCL_HEARTBEAT_TIMEOUT_SECONDS) and watchdog threads for distributed training.
• Automate node resubmission or restart logic (SUNK, Slurm, or Ray).
• Define retry thresholds to avoid infinite restart loops (MAX_RETRIES policies).
• Use Weights & Biases together with CoreWeave to push infrastructure alerts to the Weights & Biases model dashboard

Tip #4: Stress-test your infrastructure

Components don’t typically fail suddenly; they often suffer performance degradation long before a component fails. This is especially common when running longer jobs or training runs that really push the infrastructure to its max.

Before running your hero workloads, validate that everything is working with real-world stress tests and observability loops. Push the hardware to fail first with a few large test runs to figure out where the friction points exist in your cluster. 

Tips:

• Submit an NCCL test. Running these before production ensures your cluster’s InfiniBand or networking fabric can handle synchronized GPU communication without hanging or timing out.
• Continuously run validation tests, such as burn-in and burn-out of GPUs.
• Run controlled failure simulations (e.g., artificially kill one node mid-run to validate recovery behavior).
• Monitor ETTR (Effective Training Time Ratio) and MFU before and after recovery.
• Use system-wide tracing to identify hidden bottlenecks (e.g., storage latency spikes, network contention).

Practically, we understand that most teams do not want to “waste” GPU time to test recovery. However, it’s a vital step—so much so that teams at CoreWeave conduct extensive burn-in, validation, and GPU testing before handing over a cluster to a customer.

If you have an infrastructure provider, ask them what stress tests they do for you. At the very least, consider running a few NCCL tests at the application level before running a long job.

The real measure of scale: How fast you recover

In large-scale AI environments, performance and reliability are two sides of the same coin. Resilient infrastructure and systems keep your workloads running as they should, helping you get closer to predictive workloads, on-schedule deployments, and efficient utilization of your time and resources.

For more insights on improving reliability and performance at scale, take a look at these resources:

1. Webinar: How to Maximize Resiliency with AI-Native Observability
2. Checklist: Can Your Infrastructure Take the Punch?‍
3. One-pager: How CoreWeave Powers Resiliency at Scale

Whether through async checkpointing, automated recovery, predictive monitoring, or deep observability, the lesson is simple: Resiliency empowers performance at scale.

Check out the resources above or schedule time to talk with our team to learn how CoreWeave can support your AI breakthroughs.