CoreWeave: Products & Services

Best NVIDIA GPUs for Serving Inference on CoreWeave

Best NVIDIA GPUs for Serving Inference on CoreWeave

On generalized cloud infrastructures, most businesses are forced to make a decision between using a low-powered, general-purpose GPU that fits their budget, or finding the performance they need from a high-end GPU that breaks the bank. This limits certain use cases, like model serving, for companies who are looking to build products and applications using the latest developments in AI.

A major benefit of working with a specialized cloud provider is the ability to choose from a wide selection of GPUs to ensure that you are able to match the complexity of your workloads with compute that delivers an effective and scalable performance-adjusted cost. With the broadest selection of GPUs on the market, CoreWeave practically eliminates the need to choose between price or performance, empowering our clients to get the performance they need with economics that empower scale.

We encourage all of our clients to benchmark their workloads to find what works best for them. Coupled with the industry’s fastest spin-up times and most responsive auto-scaling delivered out-of-the-box with CoreWeave’s InferenceService, finding the best GPUs for the models you’re serving can reduce the amount of idle compute you consume, serve end-user demand faster in real-time, and lower inference latency while only paying for what you use. 

To help you get started, here’s some guidance on how we think about our arsenal of GPUs for model serving:

NVIDIA RTX 4000

Just because the Turing-based RTX 4000 is the smallest GPU that CoreWeave offers doesn't mean it's not cost effective. If you need to run inference for models such as the Fairseq 2.7B or GPT Neo 2.7B or smaller, this can be an excellent value for less intensive inference workloads.

However, if you are saturating the GPU with inference requests, then the more recent GPUs such as the A4000 or A5000 may serve you better. Larger contexts may require the RTX 5000, depending on how efficient your inference engine is.

NVIDIA RTX 5000

The Turing-based RTX 5000 is the smallest GPU that can run inference for the GPT-J 6B or Fairseq 6.7B models. It has double the RAM, a bit more memory bandwidth than the RTX 4000 and a much faster base clock rate.

If your 2.7B models are running out of RAM with a larger context, this is the next step up and will give you faster inference to boot.

NVIDIA A4000

The Ampere-based A4000 is a small step up from the RTX 5000, although it may not look like it at first glance. The clock rate is half that of the RTX 5000, but the boost clock nearly matches the base clock of the older GPU. What makes the difference is the number of shader cores, which is doubled.

However, the number of tensor cores is half that of the RTX 5000. Whether the A4000 or the RTX 5000 work better for your workload depends on your inference framework and what instructions you use.

NVIDIA A5000

The Ampere-based A5000 is a good step up from the A4000 and has been observed to be faster at running GPT-J 6B and Fairseq 6.7B than the A4000 for inference. It is also the smallest GPU that can be comfortably used for fine tuning smaller models, such as Fairseq, GPT Neo 1.3B or GPT Neo 2.7B.

If your model fits comfortably inside 24GB, this card is a better value proposition than the A6000. It can also host the Fairseq 13B model for inference, although it is tight at 24GB.


NVIDIA A6000

If your workload is intense enough, the Ampere-based A6000 is one of the best values for inference. It is CoreWeave's recommended GPU for fine-tuning, due to the 48GB of RAM, which allows you to fine-tune up to Fairseq 13B on a single GPU. The 48GB of RAM also allows you to batch-train steps during fine-tuning for better throughput.

The A6000 is the smallest single GPU that can host the GPT NeoX 20B model.

NVIDIA A40

Because of the value proposition, the A40 is our recommended GPU for larger-scale training jobs. The A6000 is slightly faster, but the A40 has more robust GPU drivers and more availability at CoreWeave. CoreWeave can help with setting this up. 

The A40’s 48GB of RAM allows you to batch-train steps during fine tuning for better throughput and the CoreWeave Finetuning Machine Learning Models Guide defaults to the A40 for this reason.

NVIDIA A100 40GB

The A100 40GB PCI-E nearly doubles the performance of the A40/A6000 on a single GPU basis for many workloads due to double the memory bandwidth. However, it has 8GB less RAM than the A40. This makes it difficult to host the larger models such as GPT NeoX 20B on a single GPU.

Pairs of A100 PCI-E can make excellent inference nodes if inference throughput is your primary concern. A100 NVLINK is recommended for distributed training and inference when model parallelism is required.

NVIDIA A100 80GB

With double the RAM and 30% more memory bandwidth than the A100 40GB PCI-E, this is the best GPU for large model inference on a single GPU. 20B models run as fast and comfortably on an A100 80GB PCI-E as 13B models do on an A6000.

A100 NVLINK is recommended for distributed training and inference when model parallelism is required.

The CoreWeave support team is always ready to roll-up our sleeves and help guide our clients through benchmarking workloads, maximizing our InferenceService and getting the absolute most out of the entire CoreWeave tech stack. We have a rich library of inference examples, including documentation on serving BLOOM 176B, GPT-J-6B and more. 

Get In Touch

CoreWeave is partnering with companies around the world to fuel AI & NLP development, and we’d love to help you too! Get started by speaking with one of our engineers today.


Connect with us

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.