Event details
What changes when a model actually has to handle production load?
Most inference demos run at low concurrency, where latency is stable and cost is an afterthought. Production traffic breaks that picture fast: request bursts, cache pressure, and throughput/latency tradeoffs that only show up at scale.
In this session, vLLM talks through the scheduling, memory management, and quantization decisions behind their serving stack. CoreWeave covers the infrastructure layer—networking, topology, and orchestration—that determines whether improved throughput and latency show up as real performance gains or get bottlenecked before they reach the model.
In this webinar, we'll cover:
- Why inference demos break at production scale—and how request bursts, cache pressure, and throughput/latency tradeoffs create behavior that stays hidden in stable, low-concurrency tests.
- How vLLM serves workloads at production concurrency, including scheduler and continuous-batching behavior, plus KV-cache management for long-context serving.
- How the production quantization scheme affects throughput and latency, and how to evaluate those tradeoffs alongside speculative decoding and draft-model configurations when they are part of the serving path.
- How CoreWeave infrastructure—including GB300 NVL72, networking topology, and orchestration—supports vLLM performance
- What must be true across the hardware and platform layers for scheduling and batching gains to show up as measured performance rather than being absorbed by hardware limits.
