Three things in CoreWeave AI Object Storage (CAIOS) determine how fast your GPUs read, how long your writes take, and what you pay to keep data you're not actively using: Local Object Transport Accelerator (LOTA), cross-region write acceleration, and the Archive tier. All three are configurations, not architectures, and none require rewriting an application.
Part 1 covered what each one does and why it matters. In Part 2, we demonstrate how to configure each: pointing a client at the LOTA endpoint, pre-staging a cache before the first epoch, enabling cross-region writes on a bucket, and setting an Archive policy that ages checkpoints on its own. Every example includes code you can copy and adapt: Python, boto3, or a CLI command for bucket settings.
Select a storage class and billing level
First, decide which region a bucket lives in, then choose how its data is stored and billed.
We offer a simple decision: whether you want the new Archive billing level. That comes down to two S3 API values:
Each bucket can have four storage billing levels. They relate like this:
Accelerate reads with LOTA
LOTA provides a compute-local NVMe endpoint to improve read latency. Everything else about your S3 client remains the same, and reads now come from the node-local NVMe cache when the object is present.
Three things affect how much your read latency can improve:
- object size,
- using multipart uploads, and
- pre-staging the cache before the job starts.
LOTA caches objects larger than 4 MB; anything smaller is read directly from persistent storage. For large objects, it's best practice to use multipart uploads. The object is split into parts that upload in parallel to make the best use of available bandwidth, and smaller parts limit the impact of a network error. LOTA also distributes those parts across GPU nodes so that reads are served from several nodes at once, whereas an object uploaded with a single PutObject sits on one node.
Pre-staging the LOTA cache
In the first epoch, objects are read from the bucket and stored in cache as they are initially requested. That first read pays the full bucket latency, and every miss after it adds latency you didn't need to spend.
Instead, pre-staging fills the cache with your predetermined dataset ahead of the job. This significantly improves your cache hits from the first epoch, rather than letting the ratio climb as the job runs.
To pre-stage the dataset a job will read, issue a HeadObject call with a zero-length range header against each object in it:
from botocore.client import Config
from concurrent.futures import ThreadPoolExecutor
s3 = boto3.client(
"s3",
endpoint_url="http://cwlota.com",
config=Config(
max_pool_connections=32,
s3={"addressing_style": "virtual"},
),
)
def prestage(key):
s3.head_object(
Bucket="training-data",
Key=key,
Range="bytes=0-0",
)
with ThreadPoolExecutor(max_workers=32) as pool:
list(pool.map(prestage, shard_keys))
LOTA sees each ranged head request, fetches the complete object from the bucket, and places it in the cache. No response body comes back to your client. Populate shard_keys with the objects that the job will read, and the first epoch reads local.
Three things to know:
- It only works through the LOTA endpoint: the same call against the primary endpoint returns metadata and fills nothing
- The call counts as an access: pre-staging pulls objects up into the Hot tier, which matters if you're pre-staging data that has aged down
- It's worth doing when the first-read latency is the thing holding you up: first-epoch training reads, loading model weights for inference, and checkpoint restores
Enable cross-region write acceleration
Cross-region writes are enabled on a per-bucket level between any CoreWeave region pairs.
Similar to LOTA, when cross-region writes are enabled, nothing in your application changes. The application writes to the LOTA endpoint exactly as before, and the object bytes land durably in the local region and are acknowledged at local latency. Metadata is written synchronously to the bucket's remote region, so the bucket presents a single namespace. Any reader sees the same object regardless of which region it reads. The object data migrates to the remote region in the background.
s3.upload_file(
"/tmp/checkpoint-4200.pt",
"checkpoints",
"run-17/checkpoint-4200.pt",
)
Data migration timing: migration enqueues within 24 hours and typically completes within 72 hours, depending on object size and network link conditions. This is often fine for checkpoints, since a restart reads from the local region anyway. If an object must be in the remote region by a deadline, plan for 72 hours.
Checkpoint cadence: training pauses while a checkpoint writes, so cadence has always been a tradeoff between recovery granularity and time spent training. Removing the cross-region round-trip latency from each commit removes that tradeoff. If you throttled your cadence because commit latency was high, it's worth re-testing what cadence you can now sustain.
Enable the Archive storage tier
Similar to cross-region write acceleration, Archive is enabled at a bucket level. There are two ways to utilize the new Archive storage tier: write directly to Archive or enable automated Archive on the bucket.
Option 1: Write directly to Archive. This works best if you know that an object is a long-term record rather than something you'll read soon. Writes to Archive carry additional latency compared to the Hot, Warm, and Cold billing level, so consider your application's latency tolerance.
s3.upload_file(
"/tmp/final-weights.pt",
"checkpoints",
"run-17/final-weights.pt",
ExtraArgs={"StorageClass": "STANDARD_IA"},
)
Option 2: Enable automated Archive on the bucket. Use this method when you'd rather set a rule, based on last access time, to migrate objects to the Archive tier. Two fields control this option:
# Enable automated archive after 60 days without access
cwic cwobject bucket update my-bucket \
--archive-enabled=true \
--archive-after-last-access-days=60
Reading from the Archive storage tier
Since there's no restore step, an archived object reads with a standard GET, at the same bucket and key it always had:
s3.download_file(
"checkpoints",
"run-17/final-weights.pt",
"/tmp/final-weights.pt",
)
What differs is time-to-first-byte, which is longer than the other billing levels, and overall throughput, which is lower. This is something to consider for your applications. If latency is important, you can copy the object to the STANDARD storage class ahead of a bulk read. A copy onto itself with a new storage class is the standard S3 pattern. On a versioned bucket, it creates a new version rather than replacing the existing one, which is worth accounting for in any tooling that walks version history.
s3.copy_object(
Bucket="checkpoints",
Key="run-17/final-weights.pt",
CopySource={"Bucket": "checkpoints", "Key": "run-17/final-weights.pt"},
StorageClass="STANDARD",
MetadataDirective="COPY",
)
Build a checkpoint retention policy
Let’s assume a training run in whichever region had GPUs available, writing checkpoints to a bucket hosted elsewhere. Here are the five decisions, in the order you'd make them:
Confirm it works in Mission Control
CoreWeave Mission Control® gives you visibility into your storage estate without any setup. Grafana dashboards and storage metrics are available out of the box, and the cost and CAIOS usage monitoring guide has a worked query for AI Object Storage consumption grouped by bucket.
Once automated archive is enabled, watch the Auto Archive panel group in the AI Object Storage usage panel and build your own views from the same metrics. Watch tier distribution shift as idle objects move.
Alternatively, you can use Mission Control MCP, which exposes metrics, logs, and AI Object Storage buckets as read-only tools for MCP-compatible clients. It's in public preview.
Next steps
That's the full CAIOS configuration: an endpoint change, a pre-stage pass, a per-bucket enablement, and an archive policy. Two of them you can do today; two need to be turned on for your account first.
Begin using LOTA and pre-staging today. Cross-region write acceleration and Archive are both available. Contact your account team to see how these features can improve your AI workloads.
Learn more about AI Object Storage:
- Blog Part 1: New in CoreWeave AI Object Storage: Cross-Region Writes and Archive Storage
- CoreWeave AI Object Storage (CAIOS) documentation: LOTA best practices, pre-staging, and multipart upload guidance
- On-demand webinar: How to Move Beyond Tiers, Tradeoffs, and Runaway Costs in AI Storage
